2020-03-28T05:53:34.375 INFO:root:teuthology version: 1.0.0-33ebd8f 2020-03-28T05:53:34.376 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.zYLXEA.tmp -- --verbose --lock --description rgw/multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} --name kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252 2020-03-28T05:53:34.414 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-28T05:53:34.468 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252 branch: nautilus-kefu2-testing-2020-03-27-2234 description: rgw/multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} email: tchaikov@gmail.com first_in_suite: false job_id: '4896252' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi nuke-on-error: true os_type: centos overrides: admin_socket: branch: nautilus-kefu2-testing-2020-03-27-2234 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 global: osd heartbeat grace: 40 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd fast shutdown: false osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: cd131a1eb0d13cbc9e432c4911cf5025d18aef1e valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: notcmalloc sha1: cd131a1eb0d13cbc9e432c4911cf5025d18aef1e rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: nautilus-kefu2-testing-2020-03-27-2234 sha1: cd131a1eb0d13cbc9e432c4911cf5025d18aef1e owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: cd131a1eb0d13cbc9e432c4911cf5025d18aef1e suite: rgw suite_branch: nautilus-kefu2-testing-2020-03-27-2234 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: cd131a1eb0d13cbc9e432c4911cf5025d18aef1e tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 valgrind: - --tool=memcheck - --max-threads=1024 c1.client.1: port: 8001 valgrind: - --tool=memcheck - --max-threads=1024 c2.client.0: port: 8000 valgrind: - --tool=memcheck - --max-threads=1024 c2.client.1: port: 8001 valgrind: - --tool=memcheck - --max-threads=1024 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2393 2020-03-28T05:53:34.469 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa; will attempt to use it 2020-03-28T05:53:34.469 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks 2020-03-28T05:53:34.469 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-28T05:53:34.471 INFO:teuthology.task.internal:Checking packages... 2020-03-28T05:53:34.489 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'notcmalloc' and ceph hash 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e' 2020-03-28T05:53:34.489 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-28T05:53:34.490 INFO:teuthology.packaging:ref: None 2020-03-28T05:53:34.490 INFO:teuthology.packaging:tag: None 2020-03-28T05:53:34.490 INFO:teuthology.packaging:branch: nautilus-kefu2-testing-2020-03-27-2234 2020-03-28T05:53:34.490 INFO:teuthology.packaging:sha1: cd131a1eb0d13cbc9e432c4911cf5025d18aef1e 2020-03-28T05:53:34.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&ref=nautilus-kefu2-testing-2020-03-27-2234 2020-03-28T05:53:34.729 INFO:teuthology.task.internal:Found packages for ceph version 14.2.8-334.gcd131a1eb0 2020-03-28T05:53:34.729 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-28T05:53:34.740 INFO:teuthology.task.internal:no buildpackages task found 2020-03-28T05:53:34.740 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-28T05:53:34.766 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-28T05:53:34.766 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-28T05:53:34.815 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252', 'machine_type': 'smithi'} 2020-03-28T05:53:34.833 DEBUG:teuthology.lock.ops:locked ubuntu@smithi156.front.sepia.ceph.com, ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T05:53:34.895 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-28T05:53:35.680 INFO:teuthology.provision.fog.smithi168:Scheduling deploy of centos 7.6 2020-03-28T05:53:35.693 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of centos 7.6 2020-03-28T05:53:36.403 INFO:teuthology.orchestra.console:Power off smithi168 2020-03-28T05:53:36.404 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-28T05:53:36.419 INFO:teuthology.orchestra.console:Power off smithi156 2020-03-28T05:53:36.420 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-28T05:53:36.428 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:36.439 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:36.559 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2020-03-28T05:53:36.660 INFO:teuthology.orchestra.console:Power on smithi156 2020-03-28T05:53:36.660 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-28T05:53:36.681 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:40.450 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:40.701 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:44.561 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:44.810 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:45.019 INFO:teuthology.orchestra.console:Power on for smithi156 completed 2020-03-28T05:53:45.120 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish 2020-03-28T05:53:48.672 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:48.880 INFO:teuthology.orchestra.console:Power off for smithi168 completed 2020-03-28T05:53:48.981 INFO:teuthology.orchestra.console:Power on smithi168 2020-03-28T05:53:48.981 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-28T05:53:49.000 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:53.021 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:57.134 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-28T05:53:57.343 INFO:teuthology.orchestra.console:Power on for smithi168 completed 2020-03-28T05:53:57.444 INFO:teuthology.provision.fog.smithi168:Waiting for deploy to finish 2020-03-28T05:56:17.163 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:56:26.219 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:56:35.278 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:56:44.358 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:56:53.418 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:57:02.472 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:57:11.530 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:57:20.587 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:57:29.771 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:57:38.825 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:57:47.881 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:57:56.937 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:58:05.994 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:58:15.049 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:58:24.108 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:58:26.591 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:58:26.836 INFO:teuthology.orchestra.run.smithi156:> hostname 2020-03-28T05:58:27.391 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156.front.sepia.ceph.com 2020-03-28T05:58:27.392 INFO:teuthology.orchestra.run.smithi156:> grep smithi156.front.sepia.ceph.com /etc/hosts 2020-03-28T05:58:27.479 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.15.156 smithi156.front.sepia.ceph.com smithi156 2020-03-28T05:58:27.480 INFO:teuthology.orchestra.run.smithi156:> sudo hostname smithi156 2020-03-28T05:58:27.613 INFO:teuthology.orchestra.run.smithi156:> sudo sed -i -e 's/smithi156.front.sepia.ceph.com/smithi156/g' /etc/hosts 2020-03-28T05:58:27.722 INFO:teuthology.orchestra.run.smithi156:> sudo sed -i -e 's/172.21.15.156/172.21.15.156/g' /etc/hosts 2020-03-28T05:58:27.800 INFO:teuthology.orchestra.run.smithi156:> sudo sed -i -e 's/smithi156.front.sepia.ceph.com/smithi156/g' /etc/hostname 2020-03-28T05:58:27.867 INFO:teuthology.orchestra.run.smithi156:> sudo hostname smithi156 2020-03-28T05:58:27.931 INFO:teuthology.provision.fog.smithi156:Deploy complete! 2020-03-28T05:58:27.933 DEBUG:teuthology.parallel:result is None 2020-03-28T05:58:33.194 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:58:42.253 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:59:19.882 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:59:20.157 INFO:teuthology.orchestra.run.smithi168:> hostname 2020-03-28T05:59:20.687 INFO:teuthology.orchestra.run.smithi168.stdout:smithi125 2020-03-28T05:59:20.688 INFO:teuthology.orchestra.run.smithi168:> grep smithi125 /etc/hosts 2020-03-28T05:59:20.732 INFO:teuthology.orchestra.run.smithi168.stdout:172.21.15.125 smithi125.front.sepia.ceph.com smithi125 2020-03-28T05:59:20.733 INFO:teuthology.orchestra.run.smithi168:> sudo hostname smithi168 2020-03-28T05:59:20.867 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/smithi125/smithi168/g' /etc/hosts 2020-03-28T05:59:20.925 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/172.21.15.125/172.21.15.168/g' /etc/hosts 2020-03-28T05:59:20.981 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/smithi125/smithi168/g' /etc/hostname 2020-03-28T05:59:21.033 INFO:teuthology.orchestra.run.smithi168:> sudo hostname smithi168 2020-03-28T05:59:21.089 INFO:teuthology.provision.fog.smithi168:Deploy complete! 2020-03-28T05:59:21.093 DEBUG:teuthology.parallel:result is None 2020-03-28T05:59:21.271 INFO:teuthology.lock.ops:Checking smithi156.front.sepia.ceph.com 2020-03-28T05:59:21.272 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-28T05:59:21.286 INFO:teuthology.lock.ops:Checking smithi168.front.sepia.ceph.com 2020-03-28T05:59:21.286 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-28T05:59:21.300 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: set os type and version on server 2020-03-28T05:59:21.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T05:59:21.301 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:59:21.404 INFO:teuthology.orchestra.run.smithi156:> true 2020-03-28T05:59:21.787 INFO:teuthology.orchestra.run.smithi156:> uname -m 2020-03-28T05:59:21.845 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2020-03-28T05:59:21.845 INFO:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2020-03-28T05:59:21.898 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Linux" 2020-03-28T05:59:21.898 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="7 (Core)" 2020-03-28T05:59:21.898 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2020-03-28T05:59:21.898 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2020-03-28T05:59:21.898 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="7" 2020-03-28T05:59:21.898 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-28T05:59:21.899 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2020-03-28T05:59:21.899 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-28T05:59:21.899 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.centos.org/" 2020-03-28T05:59:21.899 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-28T05:59:21.899 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:21.899 INFO:teuthology.orchestra.run.smithi156.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-28T05:59:21.899 INFO:teuthology.orchestra.run.smithi156.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-28T05:59:21.900 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-28T05:59:21.900 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-28T05:59:21.900 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:21.901 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2020-03-28T05:59:21.919 INFO:teuthology.lock.ops:Updating [smithi168.front.sepia.ceph.com]: set os type and version on server 2020-03-28T05:59:21.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T05:59:21.920 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:59:22.009 INFO:teuthology.orchestra.run.smithi168:> true 2020-03-28T05:59:22.261 INFO:teuthology.orchestra.run.smithi168:> uname -m 2020-03-28T05:59:22.302 INFO:teuthology.orchestra.run.smithi168.stdout:x86_64 2020-03-28T05:59:22.302 INFO:teuthology.orchestra.run.smithi168:> cat /etc/os-release 2020-03-28T05:59:22.382 INFO:teuthology.orchestra.run.smithi168.stdout:NAME="CentOS Linux" 2020-03-28T05:59:22.382 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION="7 (Core)" 2020-03-28T05:59:22.382 INFO:teuthology.orchestra.run.smithi168.stdout:ID="centos" 2020-03-28T05:59:22.383 INFO:teuthology.orchestra.run.smithi168.stdout:ID_LIKE="rhel fedora" 2020-03-28T05:59:22.383 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_ID="7" 2020-03-28T05:59:22.383 INFO:teuthology.orchestra.run.smithi168.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-28T05:59:22.383 INFO:teuthology.orchestra.run.smithi168.stdout:ANSI_COLOR="0;31" 2020-03-28T05:59:22.383 INFO:teuthology.orchestra.run.smithi168.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-28T05:59:22.383 INFO:teuthology.orchestra.run.smithi168.stdout:HOME_URL="https://www.centos.org/" 2020-03-28T05:59:22.384 INFO:teuthology.orchestra.run.smithi168.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-28T05:59:22.384 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T05:59:22.384 INFO:teuthology.orchestra.run.smithi168.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-28T05:59:22.384 INFO:teuthology.orchestra.run.smithi168.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-28T05:59:22.384 INFO:teuthology.orchestra.run.smithi168.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-28T05:59:22.384 INFO:teuthology.orchestra.run.smithi168.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-28T05:59:22.384 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T05:59:22.385 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2020-03-28T05:59:22.402 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-03-28T05:59:22.433 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVeY4umzW6UfyAQyF0mX5UySUYTGZr8AD8EWx3EzUiYESaXmRAx3cQSAbHtKkDkRQbTxFtAeOPifOGCXC1jA9BZHFSd8MAs/jswb2FmBxT60nVo6DtjeqWqjb4+7jAeNxo1Xj5fh+bpJKUE+XxXCWCgmtqsMHaBXN1nixUZwHtRE39ZZjfKLsAu45i122hiC9Mv7FKIhNyMSf+QqRX6jypdhrnMe8zHN2EuuFmT4GU/S7gok/a/mCWKf9FBber1PEVgW9FVmLCD26odAm6EnnA9sdVUW3cPxCxi4cYAU7/Bp7pqlmxTTC5Yir2ROfxun64//YQbMvxw/Ta765joJ9b smithi168.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKIS5Dl34X/BN+J6deeiBJbAans534cCmppZyE7VNs0qxn6GJKDt0ONQ39VC2Hu/4nXeZepysnZ2kXoNXxiERPx+Jg0CHqE8VpXRuGLpJaHfm1saOesRPQcjD1h77kMdoyPKrRG2wGtsSqtPss6hF22R5RzhCspU493aK7JRKCv/lclpkrjBrZcw+gp7Vpd1zRTj0cBB7y51kC9lYB1HjL/j9d1mulBDvqIDWwPTHw7lazcMXO7ATxHTN7v94rWJQvoJr3l6r14DqEwcIK+1b9yu+wXnE1tSpBx26S7AicxpHfVqtflDQQcgGi0/bW2/3ZKL7XsjwLaqTgKYHqvxhp 2020-03-28T05:59:22.433 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-28T05:59:22.474 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-28T05:59:22.487 INFO:teuthology.task.internal:Saving configuration 2020-03-28T05:59:22.504 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-28T05:59:22.514 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-28T05:59:22.528 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-28 05:53:34.822997', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi156.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEj/tDr5oQU5P4SDxBCjorA6Lbr9YUgq7mfjwxxUhPjqa8tbFwgMp7p8Iq2/sIPiBAvdWjySzL+SW18Ko8cI1jc=', u'description': u'/home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252'} 2020-03-28T05:59:22.542 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-28 05:53:34.822973', u'locked_by': u'scheduled_kchai@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi168.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDDjmoKX6ZgxqxsYPljkREd0J/4mCwbpHzOIF84qlQufSbZb+gCRgeA24ZUXWXX8eB96Jj4ZvZIbG5ZxAT/rIbk=', u'description': u'/home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252'} 2020-03-28T05:59:22.542 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-28T05:59:22.553 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-03-28T05:59:22.554 INFO:teuthology.task.internal:roles: ubuntu@smithi168.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-03-28T05:59:22.554 INFO:teuthology.run_tasks:Running task console_log... 2020-03-28T05:59:22.620 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-28T05:59:22.621 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-28T05:59:22.629 INFO:teuthology.task.internal:Opening connections... 2020-03-28T05:59:22.629 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T05:59:22.630 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:59:22.741 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T05:59:22.741 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T05:59:22.818 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-28T05:59:22.827 INFO:teuthology.orchestra.run.smithi156:> uname -m 2020-03-28T05:59:23.032 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2020-03-28T05:59:23.033 INFO:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2020-03-28T05:59:23.084 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Linux" 2020-03-28T05:59:23.084 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="7 (Core)" 2020-03-28T05:59:23.084 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2020-03-28T05:59:23.085 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2020-03-28T05:59:23.085 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="7" 2020-03-28T05:59:23.085 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-28T05:59:23.085 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2020-03-28T05:59:23.085 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-28T05:59:23.085 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.centos.org/" 2020-03-28T05:59:23.086 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-28T05:59:23.086 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:23.086 INFO:teuthology.orchestra.run.smithi156.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-28T05:59:23.086 INFO:teuthology.orchestra.run.smithi156.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-28T05:59:23.086 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-28T05:59:23.086 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-28T05:59:23.086 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:23.087 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2020-03-28T05:59:23.100 INFO:teuthology.orchestra.run.smithi168:> uname -m 2020-03-28T05:59:23.164 INFO:teuthology.orchestra.run.smithi168.stdout:x86_64 2020-03-28T05:59:23.164 INFO:teuthology.orchestra.run.smithi168:> cat /etc/os-release 2020-03-28T05:59:23.208 INFO:teuthology.orchestra.run.smithi168.stdout:NAME="CentOS Linux" 2020-03-28T05:59:23.208 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION="7 (Core)" 2020-03-28T05:59:23.208 INFO:teuthology.orchestra.run.smithi168.stdout:ID="centos" 2020-03-28T05:59:23.209 INFO:teuthology.orchestra.run.smithi168.stdout:ID_LIKE="rhel fedora" 2020-03-28T05:59:23.209 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_ID="7" 2020-03-28T05:59:23.209 INFO:teuthology.orchestra.run.smithi168.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-03-28T05:59:23.209 INFO:teuthology.orchestra.run.smithi168.stdout:ANSI_COLOR="0;31" 2020-03-28T05:59:23.209 INFO:teuthology.orchestra.run.smithi168.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-03-28T05:59:23.209 INFO:teuthology.orchestra.run.smithi168.stdout:HOME_URL="https://www.centos.org/" 2020-03-28T05:59:23.209 INFO:teuthology.orchestra.run.smithi168.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-28T05:59:23.210 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T05:59:23.210 INFO:teuthology.orchestra.run.smithi168.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-03-28T05:59:23.210 INFO:teuthology.orchestra.run.smithi168.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-03-28T05:59:23.210 INFO:teuthology.orchestra.run.smithi168.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-28T05:59:23.210 INFO:teuthology.orchestra.run.smithi168.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-03-28T05:59:23.210 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T05:59:23.211 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2020-03-28T05:59:23.224 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-28T05:59:23.238 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-28T05:59:23.245 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-28T05:59:23.246 INFO:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2020-03-28T05:59:23.250 INFO:teuthology.orchestra.run.smithi168:> test '!' -e /home/ubuntu/cephtest 2020-03-28T05:59:23.314 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-28T05:59:23.322 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-28T05:59:23.322 INFO:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2020-03-28T05:59:23.325 INFO:teuthology.orchestra.run.smithi168:> test -z $(ls -A /var/lib/ceph) 2020-03-28T05:59:23.409 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-28T05:59:23.455 INFO:teuthology.run_tasks:Running task kernel... 2020-03-28T05:59:23.475 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'sha1': 'distro', 'kdb': True}, 'c1.client.0': {'sha1': 'distro', 'kdb': True}, 'c1.client.1': {'sha1': 'distro', 'kdb': True}, 'c2.mon.a': {'sha1': 'distro', 'kdb': True}, 'c2.mgr.x': {'sha1': 'distro', 'kdb': True}, 'c2.client.1': {'sha1': 'distro', 'kdb': True}, 'c2.client.0': {'sha1': 'distro', 'kdb': True}, 'c2.osd.1': {'sha1': 'distro', 'kdb': True}, 'c2.osd.0': {'sha1': 'distro', 'kdb': True}, 'c2.osd.2': {'sha1': 'distro', 'kdb': True}, 'c1.osd.2': {'sha1': 'distro', 'kdb': True}, 'c1.osd.0': {'sha1': 'distro', 'kdb': True}, 'c1.osd.1': {'sha1': 'distro', 'kdb': True}, 'c1.mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-28T05:59:23.475 INFO:teuthology.task.kernel:config {'c1.mon.a': {'sha1': 'distro', 'kdb': True}, 'c2.mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-28T05:59:23.475 INFO:teuthology.orchestra.run.smithi156:> uname -r 2020-03-28T05:59:23.563 INFO:teuthology.orchestra.run.smithi156.stdout:3.10.0-1062.12.1.el7.x86_64 2020-03-28T05:59:23.564 INFO:teuthology.task.kernel:Running kernel on smithi156: 3.10.0-1062.12.1.el7.x86_64 2020-03-28T05:59:23.564 INFO:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2020-03-28T05:59:24.476 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T05:59:24.587 INFO:teuthology.orchestra.run.smithi156.stdout:Determining fastest mirrors 2020-03-28T05:59:24.871 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T05:59:24.885 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T05:59:24.887 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T05:59:24.888 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T05:59:28.328 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T05:59:28.328 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T05:59:28.329 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package kernel.x86_64 0:3.10.0-1062.18.1.el7 will be installed 2020-03-28T05:59:29.722 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T05:59:30.605 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:30.606 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T05:59:30.618 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:30.619 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T05:59:30.620 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2020-03-28T05:59:30.620 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T05:59:30.620 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T05:59:30.620 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 3.10.0-1062.18.1.el7 updates 50 M 2020-03-28T05:59:30.621 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:30.621 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T05:59:30.621 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T05:59:30.621 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2020-03-28T05:59:30.621 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T05:59:30.622 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 50 M 2020-03-28T05:59:30.622 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 64 M 2020-03-28T05:59:30.622 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T05:59:30.622 INFO:teuthology.orchestra.run.smithi156.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-28T05:59:31.614 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T05:59:31.812 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T05:59:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T05:59:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:00:08.216 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-3.10.0-1062.18.1.el7.x86_64 1/1 2020-03-28T06:00:08.777 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-3.10.0-1062.18.1.el7.x86_64 1/1 2020-03-28T06:00:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:00:08.778 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:00:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: kernel.x86_64 0:3.10.0-1062.18.1.el7 2020-03-28T06:00:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:00:08.778 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:00:08.830 INFO:teuthology.orchestra.run.smithi156:> rpm -q kernel --last | head -n 1 2020-03-28T06:00:09.163 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Sat 28 Mar 2020 05:59:35 AM UTC 2020-03-28T06:00:09.165 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:00:09.165 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.12.1.el7.x86_64 Expected: 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:00:09.166 INFO:teuthology.orchestra.run.smithi168:> uname -r 2020-03-28T06:00:09.289 INFO:teuthology.orchestra.run.smithi168.stdout:3.10.0-1062.12.1.el7.x86_64 2020-03-28T06:00:09.289 INFO:teuthology.task.kernel:Running kernel on smithi168: 3.10.0-1062.12.1.el7.x86_64 2020-03-28T06:00:09.290 INFO:teuthology.orchestra.run.smithi168:> sudo yum install -y kernel 2020-03-28T06:00:10.159 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:00:10.282 INFO:teuthology.orchestra.run.smithi168.stdout:Determining fastest mirrors 2020-03-28T06:00:17.627 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.metrocast.net 2020-03-28T06:00:17.640 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:00:17.641 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:00:17.641 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:00:20.967 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:00:20.968 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:00:20.968 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package kernel.x86_64 0:3.10.0-1062.18.1.el7 will be installed 2020-03-28T06:00:22.253 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:00:23.197 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:00:23.197 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:00:23.201 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:00:23.201 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:00:23.201 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository Size 2020-03-28T06:00:23.202 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:00:23.202 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:00:23.202 INFO:teuthology.orchestra.run.smithi168.stdout: kernel x86_64 3.10.0-1062.18.1.el7 updates 50 M 2020-03-28T06:00:23.202 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:00:23.202 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:00:23.202 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:00:23.202 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package 2020-03-28T06:00:23.203 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:00:23.203 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 50 M 2020-03-28T06:00:23.203 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 64 M 2020-03-28T06:00:23.203 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:00:23.203 INFO:teuthology.orchestra.run.smithi168.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-28T06:00:25.290 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:00:25.482 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:00:26.327 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:00:26.327 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:01:00.772 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : kernel-3.10.0-1062.18.1.el7.x86_64 1/1 2020-03-28T06:01:01.397 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : kernel-3.10.0-1062.18.1.el7.x86_64 1/1 2020-03-28T06:01:01.397 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:01:01.397 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:01:01.397 INFO:teuthology.orchestra.run.smithi168.stdout: kernel.x86_64 0:3.10.0-1062.18.1.el7 2020-03-28T06:01:01.397 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:01:01.397 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:01:01.445 INFO:teuthology.orchestra.run.smithi168:> rpm -q kernel --last | head -n 1 2020-03-28T06:01:01.797 INFO:teuthology.orchestra.run.smithi168.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Sat 28 Mar 2020 06:00:29 AM UTC 2020-03-28T06:01:01.801 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:01.801 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.12.1.el7.x86_64 Expected: 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:01.801 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-28T06:01:01.801 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-28T06:01:01.801 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-28T06:01:01.801 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-03-28T06:01:01.802 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2020-03-28T06:01:01.802 INFO:teuthology.orchestra.run.smithi156:> rpm -q kernel --last | head -n 1 2020-03-28T06:01:02.116 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Sat 28 Mar 2020 05:59:35 AM UTC 2020-03-28T06:01:02.118 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:02.118 INFO:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2020-03-28T06:01:02.267 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2020-03-28T06:01:02.267 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2020-03-28T06:01:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.02 2020-03-28T06:01:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 0.76.el7.centos 2020-03-28T06:01:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2020-03-28T06:01:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Tue 11 Feb 2020 03:34:40 PM UTC 2020-03-28T06:01:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:Group : System Environment/Base 2020-03-28T06:01:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 10080361 2020-03-28T06:01:02.268 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:11 PM UTC, Key ID 24c6a8a7f4a80eb5 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:Relocations : (not relocatable) 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : CentOS BuildSystem 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2020-03-28T06:01:02.269 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-28T06:01:02.270 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2020-03-28T06:01:02.270 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2020-03-28T06:01:02.270 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:01:02.270 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-28T06:01:02.270 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-28T06:01:02.270 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-28T06:01:02.270 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2020-03-28T06:01:02.271 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:01:02.271 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2020-03-28T06:01:02.272 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-28T06:01:02.272 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:02.272 INFO:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-28T06:01:02.765 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2020-03-28T06:01:02.877 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:02.892 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img 2020-03-28T06:01:03.134 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 2020-03-28T06:01:03.158 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img 2020-03-28T06:01:03.191 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2020-03-28T06:01:03.205 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2020-03-28T06:01:03.234 INFO:teuthology.orchestra.run.smithi156.stderr:Found linux image: /boot/vmlinuz-0-rescue-11867a3080154962a87601c586b1198d 2020-03-28T06:01:03.243 INFO:teuthology.orchestra.run.smithi156.stderr:Found initrd image: /boot/initramfs-0-rescue-11867a3080154962a87601c586b1198d.img 2020-03-28T06:01:04.358 INFO:teuthology.orchestra.run.smithi156.stderr:done 2020-03-28T06:01:04.358 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:01:04.422 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.omvMiehlwi 2020-03-28T06:01:04.422 INFO:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.omvMiehlwi 2020-03-28T06:01:04.493 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.omvMiehlwi 2020-03-28T06:01:04.622 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.omvMiehlwi is 6KB 2020-03-28T06:01:04.681 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.omvMiehlwi 2020-03-28T06:01:04.727 INFO:teuthology.orchestra.run.smithi156:> sudo grub2-set-default 0 2020-03-28T06:01:04.952 INFO:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2020-03-28T06:01:04.961 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-03-28T06:01:04.961 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi168.front.sepia.ceph.com, path=None, version=distro) 2020-03-28T06:01:04.961 INFO:teuthology.orchestra.run.smithi168:> rpm -q kernel --last | head -n 1 2020-03-28T06:01:05.337 INFO:teuthology.orchestra.run.smithi168.stdout:kernel-3.10.0-1062.18.1.el7.x86_64 Sat 28 Mar 2020 06:00:29 AM UTC 2020-03-28T06:01:05.340 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:05.340 INFO:teuthology.orchestra.run.smithi168:> sudo rpm -qi grub2-tools 2020-03-28T06:01:05.459 INFO:teuthology.orchestra.run.smithi168.stdout:Name : grub2-tools 2020-03-28T06:01:05.459 INFO:teuthology.orchestra.run.smithi168.stdout:Epoch : 1 2020-03-28T06:01:05.459 INFO:teuthology.orchestra.run.smithi168.stdout:Version : 2.02 2020-03-28T06:01:05.459 INFO:teuthology.orchestra.run.smithi168.stdout:Release : 0.76.el7.centos 2020-03-28T06:01:05.459 INFO:teuthology.orchestra.run.smithi168.stdout:Architecture: x86_64 2020-03-28T06:01:05.459 INFO:teuthology.orchestra.run.smithi168.stdout:Install Date: Tue 11 Feb 2020 03:34:40 PM UTC 2020-03-28T06:01:05.460 INFO:teuthology.orchestra.run.smithi168.stdout:Group : System Environment/Base 2020-03-28T06:01:05.460 INFO:teuthology.orchestra.run.smithi168.stdout:Size : 10080361 2020-03-28T06:01:05.460 INFO:teuthology.orchestra.run.smithi168.stdout:License : GPLv3+ 2020-03-28T06:01:05.460 INFO:teuthology.orchestra.run.smithi168.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:11 PM UTC, Key ID 24c6a8a7f4a80eb5 2020-03-28T06:01:05.460 INFO:teuthology.orchestra.run.smithi168.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2020-03-28T06:01:05.461 INFO:teuthology.orchestra.run.smithi168.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2020-03-28T06:01:05.461 INFO:teuthology.orchestra.run.smithi168.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-28T06:01:05.461 INFO:teuthology.orchestra.run.smithi168.stdout:Relocations : (not relocatable) 2020-03-28T06:01:05.461 INFO:teuthology.orchestra.run.smithi168.stdout:Packager : CentOS BuildSystem 2020-03-28T06:01:05.461 INFO:teuthology.orchestra.run.smithi168.stdout:Vendor : CentOS 2020-03-28T06:01:05.462 INFO:teuthology.orchestra.run.smithi168.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-28T06:01:05.462 INFO:teuthology.orchestra.run.smithi168.stdout:Summary : Support tools for GRUB. 2020-03-28T06:01:05.462 INFO:teuthology.orchestra.run.smithi168.stdout:Description : 2020-03-28T06:01:05.462 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:01:05.463 INFO:teuthology.orchestra.run.smithi168.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-28T06:01:05.463 INFO:teuthology.orchestra.run.smithi168.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-28T06:01:05.463 INFO:teuthology.orchestra.run.smithi168.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-28T06:01:05.463 INFO:teuthology.orchestra.run.smithi168.stdout:hardware devices. 2020-03-28T06:01:05.464 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:01:05.464 INFO:teuthology.orchestra.run.smithi168.stdout:This subpackage provides tools for support of all platforms. 2020-03-28T06:01:05.465 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-28T06:01:05.465 INFO:teuthology.task.kernel:Updating grub on smithi168 to boot 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:05.465 INFO:teuthology.orchestra.run.smithi168:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-28T06:01:05.958 INFO:teuthology.orchestra.run.smithi168.stderr:Generating grub configuration file ... 2020-03-28T06:01:06.074 INFO:teuthology.orchestra.run.smithi168.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:01:06.089 INFO:teuthology.orchestra.run.smithi168.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img 2020-03-28T06:01:06.362 INFO:teuthology.orchestra.run.smithi168.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 2020-03-28T06:01:06.382 INFO:teuthology.orchestra.run.smithi168.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img 2020-03-28T06:01:06.416 INFO:teuthology.orchestra.run.smithi168.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2020-03-28T06:01:06.437 INFO:teuthology.orchestra.run.smithi168.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2020-03-28T06:01:06.466 INFO:teuthology.orchestra.run.smithi168.stderr:Found linux image: /boot/vmlinuz-0-rescue-11867a3080154962a87601c586b1198d 2020-03-28T06:01:06.476 INFO:teuthology.orchestra.run.smithi168.stderr:Found initrd image: /boot/initramfs-0-rescue-11867a3080154962a87601c586b1198d.img 2020-03-28T06:01:07.433 INFO:teuthology.orchestra.run.smithi168.stderr:done 2020-03-28T06:01:07.435 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:01:07.505 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.8MQ9ri5G6o 2020-03-28T06:01:07.505 INFO:teuthology.orchestra.run.smithi168:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8MQ9ri5G6o 2020-03-28T06:01:07.580 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.8MQ9ri5G6o 2020-03-28T06:01:07.776 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.8MQ9ri5G6o is 6KB 2020-03-28T06:01:07.874 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.8MQ9ri5G6o 2020-03-28T06:01:07.946 INFO:teuthology.orchestra.run.smithi168:> sudo grub2-set-default 0 2020-03-28T06:01:08.130 INFO:teuthology.orchestra.run.smithi168:> sudo shutdown -r now 2020-03-28T06:01:38.161 INFO:teuthology.misc:Re-opening connections... 2020-03-28T06:01:38.161 INFO:teuthology.misc:trying to connect to ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:01:38.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:01:38.162 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:01:56.693 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T06:01:56.693 INFO:teuthology.misc:trying to connect to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:01:56.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:01:56.694 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:15.125 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.168 2020-03-28T06:02:15.125 DEBUG:teuthology.misc:waited 36.9641928673 2020-03-28T06:02:16.126 INFO:teuthology.misc:trying to connect to ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:02:16.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:16.126 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:34.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T06:02:34.581 INFO:teuthology.misc:trying to connect to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:02:34.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:34.582 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:51.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.168 2020-03-28T06:02:51.436 DEBUG:teuthology.misc:waited 73.2752709389 2020-03-28T06:02:52.436 INFO:teuthology.misc:trying to connect to ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:02:52.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:52.437 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:52.520 INFO:teuthology.orchestra.run.smithi156:> true 2020-03-28T06:02:52.806 DEBUG:teuthology.misc:waited 74.6453199387 2020-03-28T06:02:53.807 INFO:teuthology.misc:trying to connect to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:02:53.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:53.808 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:53.810 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.168 2020-03-28T06:02:53.810 DEBUG:teuthology.misc:waited 75.6490800381 2020-03-28T06:02:54.811 INFO:teuthology.misc:trying to connect to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:02:54.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:54.812 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:54.813 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.168 2020-03-28T06:02:54.813 DEBUG:teuthology.misc:waited 76.6523458958 2020-03-28T06:02:55.814 INFO:teuthology.misc:trying to connect to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:02:55.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:55.814 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:55.816 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.168 2020-03-28T06:02:55.816 DEBUG:teuthology.misc:waited 77.655050993 2020-03-28T06:02:56.817 INFO:teuthology.misc:trying to connect to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:02:56.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:56.817 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:56.819 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.168 2020-03-28T06:02:56.819 DEBUG:teuthology.misc:waited 78.6578710079 2020-03-28T06:02:57.819 INFO:teuthology.misc:trying to connect to ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:02:57.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:02:57.820 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:02:57.893 INFO:teuthology.orchestra.run.smithi168:> true 2020-03-28T06:02:58.209 DEBUG:teuthology.misc:waited 80.0482130051 2020-03-28T06:02:59.210 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-03-28T06:02:59.210 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "3.10.0-1062.18.1.el7.x86_64"... 2020-03-28T06:02:59.210 INFO:teuthology.orchestra.run.smithi156:> uname -r 2020-03-28T06:02:59.304 INFO:teuthology.orchestra.run.smithi156.stdout:3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:02:59.305 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.18.1.el7.x86_64 vs 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:02:59.305 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-28T06:02:59.305 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-03-28T06:02:59.305 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "3.10.0-1062.18.1.el7.x86_64"... 2020-03-28T06:02:59.306 INFO:teuthology.orchestra.run.smithi168:> uname -r 2020-03-28T06:02:59.377 INFO:teuthology.orchestra.run.smithi168.stdout:3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:02:59.377 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.18.1.el7.x86_64 vs 3.10.0-1062.18.1.el7.x86_64 2020-03-28T06:02:59.377 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-28T06:03:00.378 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-03-28T06:03:00.379 INFO:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-28T06:03:00.502 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2020-03-28T06:03:00.513 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-03-28T06:03:00.513 INFO:teuthology.orchestra.run.smithi168:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-28T06:03:00.597 INFO:teuthology.orchestra.run.smithi168.stdout:ttyS1 2020-03-28T06:03:00.613 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-28T06:03:00.747 INFO:teuthology.task.internal:Creating test directory... 2020-03-28T06:03:00.748 INFO:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-28T06:03:00.751 INFO:teuthology.orchestra.run.smithi168:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-28T06:03:00.805 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-28T06:03:00.879 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-28T06:03:00.966 INFO:teuthology.task.internal:Creating archive directory... 2020-03-28T06:03:00.967 INFO:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-28T06:03:00.970 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-28T06:03:01.038 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-28T06:03:01.071 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-28T06:03:01.071 INFO:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-28T06:03:01.074 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-28T06:03:01.135 INFO:teuthology.orchestra.run.smithi168.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-28T06:03:01.139 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-28T06:03:01.150 INFO:teuthology.orchestra.run.smithi168.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-28T06:03:01.153 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-28T06:03:01.155 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-28T06:03:01.165 INFO:teuthology.task.internal:Configuring sudo... 2020-03-28T06:03:01.165 INFO:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-28T06:03:01.167 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-28T06:03:01.231 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-28T06:03:01.239 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-28T06:03:01.240 INFO:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-28T06:03:01.242 INFO:teuthology.orchestra.run.smithi168:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-28T06:03:01.284 INFO:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-28T06:03:01.387 INFO:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-28T06:03:01.455 INFO:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-28T06:03:01.511 INFO:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-28T06:03:01.567 INFO:teuthology.orchestra.run.smithi156:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-28T06:03:01.620 INFO:teuthology.orchestra.run.smithi168:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-28T06:03:01.693 INFO:teuthology.orchestra.run.smithi168:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-28T06:03:01.774 INFO:teuthology.orchestra.run.smithi168:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-28T06:03:01.835 INFO:teuthology.orchestra.run.smithi168:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-28T06:03:01.912 INFO:teuthology.orchestra.run.smithi168:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-28T06:03:01.970 INFO:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2020-03-28T06:03:01.973 INFO:teuthology.orchestra.run.smithi168:> sudo service rsyslog restart 2020-03-28T06:03:02.053 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-28T06:03:02.081 INFO:teuthology.orchestra.run.smithi168.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-28T06:03:02.098 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-28T06:03:02.108 INFO:teuthology.task.internal:Starting timer... 2020-03-28T06:03:02.108 INFO:teuthology.run_tasks:Running task pcp... 2020-03-28T06:03:02.228 INFO:teuthology.run_tasks:Running task selinux... 2020-03-28T06:03:02.293 INFO:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2020-03-28T06:03:02.384 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: [ OK ] 2020-03-28T06:03:02.386 INFO:teuthology.orchestra.run.smithi168:> sudo service auditd rotate 2020-03-28T06:03:02.475 INFO:teuthology.orchestra.run.smithi168.stdout:Rotating logs: [ OK ] 2020-03-28T06:03:02.477 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-28T06:03:02.477 INFO:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2020-03-28T06:03:02.534 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2020-03-28T06:03:02.535 INFO:teuthology.orchestra.run.smithi168:> /usr/sbin/getenforce 2020-03-28T06:03:02.582 INFO:teuthology.orchestra.run.smithi168.stdout:Permissive 2020-03-28T06:03:02.582 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi156.front.sepia.ceph.com': u'permissive', 'ubuntu@smithi168.front.sepia.ceph.com': u'permissive'} 2020-03-28T06:03:02.583 INFO:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\)' 2020-03-28T06:03:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-28T06:03:02.636 INFO:teuthology.orchestra.run.smithi168:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\)' 2020-03-28T06:03:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-28T06:03:02.692 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-28T06:03:02.692 INFO:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2020-03-28T06:03:02.823 INFO:teuthology.orchestra.run.smithi168:> sudo /usr/sbin/setenforce permissive 2020-03-28T06:03:02.879 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-28T06:03:02.887 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-28T06:03:02.888 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-28T06:03:02.904 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-28T06:03:02.906 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi156.front.sepia.ceph.com,smithi168.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-28T06:03:03.246 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-28T06:03:04.351 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-28T06:03:07.235 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-28T06:03:07.258 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:07 +0000 (0:00:00.060) 0:00:00.061 ******** 2020-03-28T06:03:07.271 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:07 +0000 (0:00:00.012) 0:00:00.073 ******** 2020-03-28T06:03:10.170 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi168.front.sepia.ceph.com] 2020-03-28T06:03:10.243 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:10 +0000 (0:00:02.972) 0:00:03.046 ******** 2020-03-28T06:03:10.530 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] 2020-03-28T06:03:10.576 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-28T06:03:10.577 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-28T06:03:10.608 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:10 +0000 (0:00:00.364) 0:00:03.410 ******** 2020-03-28T06:03:10.635 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:10 +0000 (0:00:00.027) 0:00:03.438 ******** 2020-03-28T06:03:10.920 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-28T06:03:10.958 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:10 +0000 (0:00:00.322) 0:00:03.760 ******** 2020-03-28T06:03:11.339 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-28T06:03:11.340 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-28T06:03:11.379 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:11 +0000 (0:00:00.420) 0:00:04.181 ******** 2020-03-28T06:03:11.385 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-28T06:03:11.423 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:11 +0000 (0:00:00.044) 0:00:04.225 ******** 2020-03-28T06:03:11.428 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-28T06:03:11.429 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:11.466 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:11 +0000 (0:00:00.043) 0:00:04.269 ******** 2020-03-28T06:03:11.470 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:11.508 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:11 +0000 (0:00:00.042) 0:00:04.311 ******** 2020-03-28T06:03:11.879 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-28T06:03:11.881 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:03:11.926 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:11 +0000 (0:00:00.418) 0:00:04.729 ******** 2020-03-28T06:03:11.944 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:03:11.983 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:11 +0000 (0:00:00.056) 0:00:04.786 ******** 2020-03-28T06:03:13.398 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-28T06:03:13.400 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375391.94-225120479159630/source", "state": "file", "uid": 0} 2020-03-28T06:03:13.441 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:13 +0000 (0:00:01.458) 0:00:06.244 ******** 2020-03-28T06:03:13.447 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375392.0-82921181574790/source", "state": "file", "uid": 0} 2020-03-28T06:03:13.485 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:13 +0000 (0:00:00.043) 0:00:06.288 ******** 2020-03-28T06:03:13.986 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-28T06:03:13.987 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-28T06:03:13.989 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-28T06:03:14.026 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlC 2020-03-28T06:03:14.028 INFO:teuthology.task.ansible.out:yoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, 2020-03-28T06:03:14.029 INFO:teuthology.task.ansible.out:"state": "present", "user": "cm", "validate_certs": true} 2020-03-28T06:03:14.143 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-28T06:03:14.150 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-28T06:03:14.343 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:14 +0000 (0:00:00.857) 0:00:07.145 ******** 2020-03-28T06:03:14.376 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:14 +0000 (0:00:00.032) 0:00:07.178 ******** 2020-03-28T06:03:15.175 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi156.front.sepia.ceph.com] 2020-03-28T06:03:15.265 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-28T06:03:15.266 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:15 +0000 (0:00:00.888) 0:00:08.067 ******** 2020-03-28T06:03:15.277 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] 2020-03-28T06:03:15.369 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:15 +0000 (0:00:00.104) 0:00:08.171 ******** 2020-03-28T06:03:15.883 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-28T06:03:15.884 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.222788", "end": "2020-03-28 06:03:15.860709", "rc": 0, "start": "2020-03-28 06:03:15.637921", "stderr": "", "stderr_lines": [], "stdout": "02eae6d5814ccf641ed00343442d83bf2a286582", "stdout_lines": ["02eae6d5814ccf641ed00343442d83bf2a286582"]} changed: [smithi168.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.233530", "end": "2020-03-28 06:03:15.864943", "rc": 0, "start": "2020-03-28 06:03:15.631413", "stderr": "", "stderr_lines": [], "stdout": "02eae6d5814ccf641ed00343442d83bf2a286582", "stdout_lines": ["02eae6d5814ccf641ed00343442d83bf2a286582"]} 2020-03-28T06:03:15.920 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:15 +0000 (0:00:00.550) 0:00:08.722 ******** 2020-03-28T06:03:15.952 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:15 +0000 (0:00:00.032) 0:00:08.754 ******** 2020-03-28T06:03:16.246 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-28T06:03:16.247 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009585", "end": "2020-03-28 06:03:16.220137", "failed_when_result": false, "rc": 0, "start": "2020-03-28 06:03:16.210552", "stderr": "", "stderr_lines": [], "stdout": "966e9a6bfb22958c64d4123b47ddee87a2318d5a", "stdout_lines": ["966e9a6bfb22958c64d4123b47ddee87a2318d5a"]} 2020-03-28T06:03:16.248 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012634", "end": "2020-03-28 06:03:16.226290", "failed_when_result": false, "rc": 0, "start": "2020-03-28 06:03:16.213656", "stderr": "", "stderr_lines": [], "stdout": "966e9a6bfb22958c64d4123b47ddee87a2318d5a", "stdout_lines": ["966e9a6bfb22958c64d4123b47ddee87a2318d5a"]} 2020-03-28T06:03:16.285 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.333) 0:00:09.088 ******** 2020-03-28T06:03:16.341 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.049) 0:00:09.137 ******** TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.380 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.044) 0:00:09.182 ******** 2020-03-28T06:03:16.387 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.428 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.047) 0:00:09.230 ******** 2020-03-28T06:03:16.434 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-28T06:03:16.434 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.487 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.059) 0:00:09.290 ******** 2020-03-28T06:03:16.493 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.535 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.047) 0:00:09.337 ******** 2020-03-28T06:03:16.539 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-28T06:03:16.539 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.581 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.046) 0:00:09.383 ******** 2020-03-28T06:03:16.586 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.625 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.044) 0:00:09.427 ******** 2020-03-28T06:03:16.632 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-28T06:03:16.634 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.669 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.043) 0:00:09.471 ******** 2020-03-28T06:03:16.675 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.711 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.042) 0:00:09.513 ******** 2020-03-28T06:03:16.716 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-28T06:03:16.718 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.756 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.044) 0:00:09.558 ******** 2020-03-28T06:03:16.762 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:16.798 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:16 +0000 (0:00:00.042) 0:00:09.601 ******** 2020-03-28T06:03:17.095 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-28T06:03:17.096 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:03:17.141 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:17 +0000 (0:00:00.342) 0:00:09.943 ******** 2020-03-28T06:03:17.150 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:03:17.188 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:17 +0000 (0:00:00.045) 0:00:09.989 ******** 2020-03-28T06:03:17.538 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-28T06:03:17.539 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-03-28T06:03:17.575 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-03-28T06:03:17.857 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-03-28T06:03:17.912 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-03-28T06:03:18.216 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-03-28T06:03:18.238 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-03-28T06:03:18.519 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-03-28T06:03:18.562 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-03-28T06:03:18.850 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-03-28T06:03:18.904 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-03-28T06:03:19.198 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-03-28T06:03:19.252 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-03-28T06:03:19.548 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-03-28T06:03:19.549 INFO:teuthology.task.ansible.out: 2020-03-28T06:03:19.601 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-03-28T06:03:20.153 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9', u'name': u'jcaratzas'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-03-28T06:03:20.253 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9', u'name': u'jcaratzas'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-03-28T06:03:20.432 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-03-28T06:03:20.605 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-03-28T06:03:20.733 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-03-28T06:03:20.885 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-03-28T06:03:21.085 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-03-28T06:03:21.212 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-03-28T06:03:21.383 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-03-28T06:03:21.556 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-03-28T06:03:21.726 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-03-28T06:03:21.893 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-03-28T06:03:22.055 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-03-28T06:03:22.182 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-03-28T06:03:22.364 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-03-28T06:03:22.526 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-03-28T06:03:22.673 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-03-28T06:03:22.845 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-03-28T06:03:22.963 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-03-28T06:03:23.193 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-03-28T06:03:23.290 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-03-28T06:03:23.553 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-03-28T06:03:23.847 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed', u'name': u'cmeno'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "name": "cmeno", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-03-28T06:03:24.084 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed', u'name': u'cmeno'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "name": "cmeno", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-03-28T06:03:24.189 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-03-28T06:03:24.422 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-03-28T06:03:24.516 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-03-28T06:03:24.748 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-03-28T06:03:24.822 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-03-28T06:03:25.072 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-03-28T06:03:25.125 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-03-28T06:03:25.385 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-03-28T06:03:25.444 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-03-28T06:03:25.728 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-03-28T06:03:25.763 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-03-28T06:03:26.082 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-03-28T06:03:26.092 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-03-28T06:03:26.413 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-03-28T06:03:26.429 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-03-28T06:03:26.716 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-03-28T06:03:26.749 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-03-28T06:03:27.020 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-03-28T06:03:27.084 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-03-28T06:03:27.307 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-03-28T06:03:27.388 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-03-28T06:03:27.627 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-03-28T06:03:27.732 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-03-28T06:03:27.919 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-03-28T06:03:28.040 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-03-28T06:03:28.227 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-03-28T06:03:28.374 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-03-28T06:03:28.511 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-03-28T06:03:28.720 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-03-28T06:03:28.794 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-03-28T06:03:29.069 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-03-28T06:03:29.123 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-03-28T06:03:29.419 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-03-28T06:03:29.452 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-03-28T06:03:29.763 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-03-28T06:03:29.776 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-03-28T06:03:30.082 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-03-28T06:03:30.094 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-03-28T06:03:30.407 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-03-28T06:03:30.440 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-03-28T06:03:30.728 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-28T06:03:30.782 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-28T06:03:31.037 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-03-28T06:03:31.080 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-03-28T06:03:31.330 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-03-28T06:03:31.363 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-03-28T06:03:31.633 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-03-28T06:03:31.697 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-03-28T06:03:31.990 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-03-28T06:03:32.001 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-03-28T06:03:32.321 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-03-28T06:03:32.353 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-03-28T06:03:32.658 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-03-28T06:03:32.660 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-03-28T06:03:32.922 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-03-28T06:03:32.986 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-03-28T06:03:33.200 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-28T06:03:33.306 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-28T06:03:33.494 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-03-28T06:03:33.625 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-03-28T06:03:33.837 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-03-28T06:03:33.973 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-03-28T06:03:34.187 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-03-28T06:03:34.319 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-03-28T06:03:34.538 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-03-28T06:03:34.644 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-03-28T06:03:34.833 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-03-28T06:03:34.973 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-03-28T06:03:35.131 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-03-28T06:03:35.289 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-03-28T06:03:35.426 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-03-28T06:03:35.651 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-03-28T06:03:35.738 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-03-28T06:03:35.982 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-03-28T06:03:36.059 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-03-28T06:03:36.304 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-03-28T06:03:36.400 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-03-28T06:03:36.631 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-03-28T06:03:36.751 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-03-28T06:03:36.957 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-03-28T06:03:37.083 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-03-28T06:03:37.293 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-03-28T06:03:37.418 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-03-28T06:03:37.639 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-03-28T06:03:37.741 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-03-28T06:03:37.972 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-03-28T06:03:38.083 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-03-28T06:03:38.326 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-03-28T06:03:38.401 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-03-28T06:03:38.679 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-03-28T06:03:38.712 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-28T06:03:39.028 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-03-28T06:03:39.073 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-03-28T06:03:39.394 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-03-28T06:03:39.401 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-03-28T06:03:39.730 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-03-28T06:03:39.743 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-03-28T06:03:40.070 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-03-28T06:03:40.103 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-03-28T06:03:40.384 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-03-28T06:03:40.447 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-03-28T06:03:40.743 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-03-28T06:03:40.810 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-03-28T06:03:41.101 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-03-28T06:03:41.164 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-03-28T06:03:41.439 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-03-28T06:03:41.502 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-03-28T06:03:41.811 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-03-28T06:03:41.864 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-03-28T06:03:42.134 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-03-28T06:03:42.200 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-03-28T06:03:42.464 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-03-28T06:03:42.519 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-03-28T06:03:42.787 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-03-28T06:03:42.844 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-03-28T06:03:43.129 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-03-28T06:03:43.158 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-03-28T06:03:43.462 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-03-28T06:03:43.522 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-03-28T06:03:43.799 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-03-28T06:03:43.853 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-03-28T06:03:44.126 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-03-28T06:03:44.200 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-03-28T06:03:44.450 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-03-28T06:03:44.523 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-03-28T06:03:44.759 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-03-28T06:03:44.832 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-03-28T06:03:45.097 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-03-28T06:03:45.171 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-03-28T06:03:45.394 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-03-28T06:03:45.528 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-03-28T06:03:45.724 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-03-28T06:03:45.854 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-03-28T06:03:46.010 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-03-28T06:03:46.234 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-03-28T06:03:46.329 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-03-28T06:03:46.573 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-03-28T06:03:46.667 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-03-28T06:03:46.920 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-03-28T06:03:47.136 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-03-28T06:03:47.254 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-03-28T06:03:47.421 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-03-28T06:03:47.710 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-03-28T06:03:47.744 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-03-28T06:03:48.061 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-03-28T06:03:48.062 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-03-28T06:03:48.366 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-03-28T06:03:48.404 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-03-28T06:03:48.724 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-03-28T06:03:48.728 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-03-28T06:03:49.040 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-03-28T06:03:49.096 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-03-28T06:03:49.098 INFO:teuthology.task.ansible.out: 2020-03-28T06:03:49.317 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-03-28T06:03:49.433 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-03-28T06:03:49.682 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-03-28T06:03:49.791 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-03-28T06:03:50.039 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-03-28T06:03:50.144 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-03-28T06:03:50.376 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-03-28T06:03:50.439 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-03-28T06:03:50.739 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-03-28T06:03:50.772 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-03-28T06:03:51.087 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-03-28T06:03:51.134 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-03-28T06:03:51.442 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-03-28T06:03:51.466 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-28T06:03:51.809 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-03-28T06:03:51.852 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-03-28T06:03:52.160 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-03-28T06:03:52.162 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-03-28T06:03:52.497 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-03-28T06:03:52.793 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-03-28T06:03:52.817 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-03-28T06:03:53.114 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-03-28T06:03:53.157 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-03-28T06:03:53.422 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-03-28T06:03:53.502 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-03-28T06:03:53.749 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-03-28T06:03:53.835 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-03-28T06:03:54.101 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-03-28T06:03:54.173 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-03-28T06:03:54.433 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-03-28T06:03:54.761 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "name": "khiremat", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-03-28T06:03:54.782 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-03-28T06:03:55.381 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "name": "khiremat", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-03-28T06:03:55.383 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}, "name": "kapandya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2020-03-28T06:03:55.515 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:55 +0000 (0:00:38.328) 0:00:48.318 ******** 2020-03-28T06:03:55.546 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-28T06:03:55.582 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:55 +0000 (0:00:00.066) 0:00:48.385 ******** 2020-03-28T06:03:55.640 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c 2020-03-28T06:03:55.641 INFO:teuthology.task.ansible.out:68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwn 2020-03-28T06:03:55.643 INFO:teuthology.task.ansible.out:j7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2 2020-03-28T06:03:55.644 INFO:teuthology.task.ansible.out:714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat. 2020-03-28T06:03:55.645 INFO:teuthology.task.ansible.out:com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea856167 2020-03-28T06:03:55.647 INFO:teuthology.task.ansible.out:77a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c14 2020-03-28T06:03:55.648 INFO:teuthology.task.ansible.out:9791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabr 2020-03-28T06:03:55.649 INFO:teuthology.task.ansible.out:ioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaa 2020-03-28T06:03:55.651 INFO:teuthology.task.ansible.out:rit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields I 2020-03-28T06:03:55.652 INFO:teuthology.task.ansible.out:Zq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db 2020-03-28T06:03:55.655 INFO:teuthology.task.ansible.out:4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iO 2020-03-28T06:03:55.656 INFO:teuthology.task.ansible.out:J68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}]}, "changed": false} 2020-03-28T06:03:55.698 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:55 +0000 (0:00:00.115) 0:00:48.500 ******** 2020-03-28T06:03:55.987 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}, "name": "kapandya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2020-03-28T06:03:56.165 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:56 +0000 (0:00:00.461) 0:00:48.961 ******** 2020-03-28T06:03:56.186 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-28T06:03:56.232 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:56 +0000 (0:00:00.073) 0:00:49.035 ******** 2020-03-28T06:03:56.297 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-28T06:03:56.300 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-03-28T06:03:56.301 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAx 2020-03-28T06:03:56.302 INFO:teuthology.task.ansible.out:j+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8ea 2020-03-28T06:03:56.304 INFO:teuthology.task.ansible.out:c8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f2610 2020-03-28T06:03:56.305 INFO:teuthology.task.ansible.out:2d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwn 2020-03-28T06:03:56.305 INFO:teuthology.task.ansible.out:skQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiw 2020-03-28T06:03:56.307 INFO:teuthology.task.ansible.out:ang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat 2020-03-28T06:03:56.308 INFO:teuthology.task.ansible.out:.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "p 2020-03-28T06:03:56.310 INFO:teuthology.task.ansible.out:cuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, 2020-03-28T06:03:56.311 INFO:teuthology.task.ansible.out:{"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b3 2020-03-28T06:03:56.312 INFO:teuthology.task.ansible.out:9b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49f 2020-03-28T06:03:56.314 INFO:teuthology.task.ansible.out:b166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}]}, "changed": false} 2020-03-28T06:03:56.314 INFO:teuthology.task.ansible.out: 2020-03-28T06:03:56.362 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:56 +0000 (0:00:00.129) 0:00:49.164 ******** 2020-03-28T06:03:56.705 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-28T06:03:56.706 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com -> localhost] => {"after": "02eae6d5814ccf641ed00343442d83bf2a286582", "attempts": 1, "before": "02eae6d5814ccf641ed00343442d83bf2a286582", "changed": false, "remote_url_changed": false} 2020-03-28T06:03:56.743 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:56 +0000 (0:00:00.380) 0:00:49.545 ******** 2020-03-28T06:03:57.108 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com -> localhost] => {"after": "02eae6d5814ccf641ed00343442d83bf2a286582", "attempts": 1, "before": "02eae6d5814ccf641ed00343442d83bf2a286582", "changed": false, "remote_url_changed": false} 2020-03-28T06:03:57.146 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:03:57 +0000 (0:00:00.403) 0:00:49.948 ******** 2020-03-28T06:03:57.152 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-28T06:03:57.153 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-03-28T06:03:57.477 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-03-28T06:03:57.543 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-03-28T06:03:57.846 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-03-28T06:03:57.898 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-03-28T06:03:58.155 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNK 2020-03-28T06:03:58.157 INFO:teuthology.task.ansible.out:MqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-03-28T06:03:58.250 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-03-28T06:03:58.504 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-03-28T06:03:58.593 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNK 2020-03-28T06:03:58.595 INFO:teuthology.task.ansible.out:MqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-03-28T06:03:58.857 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-03-28T06:03:58.859 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-03-28T06:03:58.957 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-03-28T06:03:59.182 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-03-28T06:03:59.276 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j 2020-03-28T06:03:59.277 INFO:teuthology.task.ansible.out:7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-03-28T06:03:59.511 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9', u'name': u'jcaratzas'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-03-28T06:03:59.631 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-03-28T06:03:59.854 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-03-28T06:03:59.865 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:03:59.982 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9', u'name': u'jcaratzas'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-03-28T06:04:00.213 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-03-28T06:04:00.215 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-03-28T06:04:00.225 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:00.236 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:00.248 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:00.338 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-03-28T06:04:00.371 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:00.608 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-03-28T06:04:00.752 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-03-28T06:04:00.754 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-03-28T06:04:00.764 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:00.765 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:00.776 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:00.906 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-03-28T06:04:00.907 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-03-28T06:04:01.140 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-03-28T06:04:01.250 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-03-28T06:04:01.476 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-03-28T06:04:01.478 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-03-28T06:04:01.560 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-03-28T06:04:01.816 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-03-28T06:04:01.869 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed', u'name': u'cmeno'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-03-28T06:04:02.157 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-03-28T06:04:02.190 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-03-28T06:04:02.509 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed', u'name': u'cmeno'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-03-28T06:04:02.521 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-03-28T06:04:02.806 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-03-28T06:04:02.881 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-03-28T06:04:03.158 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-03-28T06:04:03.246 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-03-28T06:04:03.248 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-03-28T06:04:03.494 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-03-28T06:04:03.571 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-03-28T06:04:03.573 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-03-28T06:04:03.863 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-03-28T06:04:03.865 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-03-28T06:04:03.917 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-03-28T06:04:04.193 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-03-28T06:04:04.195 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-03-28T06:04:04.236 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-03-28T06:04:04.537 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-03-28T06:04:04.556 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-03-28T06:04:04.558 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-03-28T06:04:04.891 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-03-28T06:04:04.910 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-03-28T06:04:05.257 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-03-28T06:04:05.259 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-03-28T06:04:05.268 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-03-28T06:04:05.640 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-03-28T06:04:05.653 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-03-28T06:04:05.966 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-03-28T06:04:05.974 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-03-28T06:04:06.305 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-03-28T06:04:06.305 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-03-28T06:04:06.641 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-03-28T06:04:06.642 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-03-28T06:04:06.663 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-03-28T06:04:07.145 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-03-28T06:04:07.146 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-03-28T06:04:07.475 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-03-28T06:04:07.477 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-03-28T06:04:07.487 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-03-28T06:04:07.825 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-03-28T06:04:07.829 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-03-28T06:04:08.171 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-03-28T06:04:08.175 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-03-28T06:04:08.507 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-03-28T06:04:08.522 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": tr 2020-03-28T06:04:08.523 INFO:teuthology.task.ansible.out:ue} 2020-03-28T06:04:08.827 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-03-28T06:04:08.866 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-03-28T06:04:09.185 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2Gu 2020-03-28T06:04:09.187 INFO:teuthology.task.ansible.out:FUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-03-28T06:04:09.490 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-03-28T06:04:09.543 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-03-28T06:04:09.839 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-03-28T06:04:09.888 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-28T06:04:10.184 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-03-28T06:04:10.186 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-03-28T06:04:10.239 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-03-28T06:04:10.536 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-03-28T06:04:10.537 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-03-28T06:04:10.599 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-03-28T06:04:10.882 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-03-28T06:04:10.949 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-03-28T06:04:10.951 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-03-28T06:04:11.231 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-03-28T06:04:11.293 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-03-28T06:04:11.295 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-03-28T06:04:11.553 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-03-28T06:04:11.648 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-03-28T06:04:11.898 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-03-28T06:04:11.900 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-28T06:04:11.998 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-03-28T06:04:12.256 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-03-28T06:04:12.257 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-03-28T06:04:12.259 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-03-28T06:04:12.369 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-03-28T06:04:12.616 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-03-28T06:04:12.712 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-03-28T06:04:12.773 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-28T06:04:12.960 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-03-28T06:04:13.090 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-03-28T06:04:13.091 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-03-28T06:04:13.093 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-03-28T06:04:13.312 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-03-28T06:04:13.440 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-03-28T06:04:13.648 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-03-28T06:04:13.790 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-03-28T06:04:13.973 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-03-28T06:04:14.160 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-03-28T06:04:14.306 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-03-28T06:04:14.466 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-03-28T06:04:14.632 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-03-28T06:04:14.815 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-03-28T06:04:14.983 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-03-28T06:04:15.153 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-03-28T06:04:15.335 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-03-28T06:04:15.337 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-03-28T06:04:15.481 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-03-28T06:04:15.668 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-03-28T06:04:15.826 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-03-28T06:04:16.011 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-03-28T06:04:16.182 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-03-28T06:04:16.183 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-03-28T06:04:16.381 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-03-28T06:04:16.382 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-03-28T06:04:16.517 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-03-28T06:04:16.681 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-03-28T06:04:16.828 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-03-28T06:04:16.992 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-03-28T06:04:17.214 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-03-28T06:04:17.215 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-03-28T06:04:17.368 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-03-28T06:04:17.569 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-03-28T06:04:17.686 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-28T06:04:17.933 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-03-28T06:04:18.016 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-03-28T06:04:18.256 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-03-28T06:04:18.380 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-03-28T06:04:18.609 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-03-28T06:04:18.736 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-03-28T06:04:18.964 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-03-28T06:04:19.079 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-03-28T06:04:19.305 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-03-28T06:04:19.447 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-03-28T06:04:19.659 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-03-28T06:04:19.785 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-03-28T06:04:19.997 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-03-28T06:04:20.134 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-03-28T06:04:20.380 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-03-28T06:04:20.480 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-03-28T06:04:20.690 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-03-28T06:04:20.826 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-03-28T06:04:21.048 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-03-28T06:04:21.179 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-03-28T06:04:21.412 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-03-28T06:04:21.518 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-03-28T06:04:21.738 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-03-28T06:04:21.855 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-03-28T06:04:22.096 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-03-28T06:04:22.221 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-03-28T06:04:22.446 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-03-28T06:04:22.557 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-03-28T06:04:22.799 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-03-28T06:04:22.906 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-03-28T06:04:23.153 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-03-28T06:04:23.247 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-03-28T06:04:23.498 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-03-28T06:04:23.583 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-03-28T06:04:23.822 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-03-28T06:04:23.924 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-03-28T06:04:24.167 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-03-28T06:04:24.257 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-03-28T06:04:24.515 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-03-28T06:04:24.608 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-03-28T06:04:24.862 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-03-28T06:04:24.990 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-03-28T06:04:25.194 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-03-28T06:04:25.330 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-03-28T06:04:25.332 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-03-28T06:04:25.506 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-03-28T06:04:25.701 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-03-28T06:04:25.703 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-03-28T06:04:25.874 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-03-28T06:04:25.876 INFO:teuthology.task.ansible.out: 2020-03-28T06:04:26.014 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-03-28T06:04:26.233 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-03-28T06:04:26.245 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-03-28T06:04:26.364 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-03-28T06:04:26.602 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-03-28T06:04:26.603 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-03-28T06:04:26.694 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-03-28T06:04:26.933 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-03-28T06:04:27.016 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-03-28T06:04:27.292 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-03-28T06:04:27.379 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-03-28T06:04:27.637 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-03-28T06:04:27.690 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-03-28T06:04:27.982 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-03-28T06:04:28.035 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-03-28T06:04:28.327 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-03-28T06:04:28.400 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-03-28T06:04:28.684 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-03-28T06:04:28.727 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-03-28T06:04:29.030 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-03-28T06:04:29.062 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-03-28T06:04:29.396 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-03-28T06:04:29.415 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-03-28T06:04:29.751 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-03-28T06:04:29.789 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-03-28T06:04:30.104 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-03-28T06:04:30.118 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-03-28T06:04:30.456 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-03-28T06:04:30.470 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-03-28T06:04:30.805 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-03-28T06:04:30.807 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-28T06:04:30.827 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-03-28T06:04:31.137 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-03-28T06:04:31.201 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-03-28T06:04:31.493 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-03-28T06:04:31.525 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-03-28T06:04:31.852 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-03-28T06:04:31.866 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-03-28T06:04:31.867 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-03-28T06:04:32.212 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-03-28T06:04:32.223 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-03-28T06:04:32.562 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-03-28T06:04:32.604 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-03-28T06:04:32.937 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-03-28T06:04:32.939 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-03-28T06:04:33.282 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-03-28T06:04:33.284 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-03-28T06:04:33.614 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-03-28T06:04:33.615 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-03-28T06:04:33.937 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-03-28T06:04:33.968 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-03-28T06:04:34.283 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-03-28T06:04:34.329 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpzfLLhjb6WlU3D0D26KAzcSgy9FkG1jnU2X6PqvyGuxyqrfDfLRc6d9A/clStS6GmyKfmevMLFr0Qn17FbPguZJ98Cnx4WwTyRyDSY6M4aoHdhKJgX6zGF7EJxCgb4gJdzlVO8uWD9k1ULV5ShS6z24rkjmxtX8Q/nkDa7w6XThLXQHPXmEJZ7ZsURaMwU+A5K0Mun/RO9W1TRKBzqspvb8D8q/o5R1T4Y6nSgMI8c3fbiuikaMogbPFTzjp154VNBD08kqTpi6JFgWUbhp6Cgpi9NJprsaPLvf1hOBnPvMENy+hkLE7TKnOqFPW6NglbB7igLOlLdqZgNfalX+XSkklZgqrJX/8/eXgGAKE4AvPDerAbBd2X9H2I5P84o5Rs8cDw2vmV5zPPT0SOJP+Ge8eXanekbVxbNI3UrrEEj/o3vQz2/ng1EpknlySpy8FIEDKF1qZ/M9Gku0RcquEZTYsUlGkxl1RYh6VhSp2W6QSeAl4B8CUedv4zQ0Cqr18= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-03-28T06:04:34.438 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:34 +0000 (0:00:37.292) 0:01:27.241 ******** 2020-03-28T06:04:34.486 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-28T06:04:34.487 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.497 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.498 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.509 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.510 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.521 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.533 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.544 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9', u'name': u'jcaratzas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.643 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-28T06:04:34.644 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-03-28T06:04:34.688 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-28T06:04:34.690 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:34.976 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-28T06:04:34.978 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-03-28T06:04:35.020 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-28T06:04:35.021 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-03-28T06:04:35.022 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.324 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpzfLLhjb6WlU3D0D26KAzcSgy9FkG1jnU2X6PqvyGuxyqrfDfLRc6d9A/clStS6GmyKfmevMLFr0Qn17FbPguZJ98Cnx4WwTyRyDSY6M4aoHdhKJgX6zGF7EJxCgb4gJdzlVO8uWD9k1ULV5ShS6z24rkjmxtX8Q/nkDa7w6XThLXQHPXmEJZ7ZsURaMwU+A5K0Mun/RO9W1TRKBzqspvb8D8q/o5R1T4Y6nSgMI8c3fbiuikaMogbPFTzjp154VNBD08kqTpi6JFgWUbhp6Cgpi9NJprsaPLvf1hOBnPvMENy+hkLE7TKnOqFPW6NglbB7igLOlLdqZgNfalX+XSkklZgqrJX/8/eXgGAKE4AvPDerAbBd2X9H2I5P84o5Rs8cDw2vmV5zPPT0SOJP+Ge8eXanekbVxbNI3UrrEEj/o3vQz2/ng1EpknlySpy8FIEDKF1qZ/M9Gku0RcquEZTYsUlGkxl1RYh6VhSp2W6QSeAl4B8CUedv4zQ0Cqr18= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-03-28T06:04:35.401 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-03-28T06:04:35.437 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:35 +0000 (0:00:00.998) 0:01:28.240 ******** 2020-03-28T06:04:35.611 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.622 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.633 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.634 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.645 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.656 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.669 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.683 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9', u'name': u'jcaratzas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:35.726 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-03-28T06:04:36.032 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-03-28T06:04:36.054 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-03-28T06:04:36.055 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.066 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.078 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.091 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.109 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.120 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed', u'name': u'cmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.156 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.157 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.158 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.160 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.170 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.171 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.197 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.209 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.210 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.222 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.234 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.258 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.259 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.270 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.292 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.293 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.295 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.304 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.315 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.328 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.329 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.340 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.351 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.373 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.476 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.487 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-03-28T06:04:36.488 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.489 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.491 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loo 2020-03-28T06:04:36.493 INFO:teuthology.task.ansible.out:p_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.494 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.526 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.527 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.530 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.539 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.550 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.561 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.563 INFO:teuthology.task.ansible.out: skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.577 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.585 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.596 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.607 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.618 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.629 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.629 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.642 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.658 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.666 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.677 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.688 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.689 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.704 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.711 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.734 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.747 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0X 2020-03-28T06:04:36.748 INFO:teuthology.task.ansible.out:lFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.758 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.770 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.771 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": 2020-03-28T06:04:36.771 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-03-28T06:04:36.782 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.815 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.815 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.817 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.828 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.849 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.863 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.870 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.871 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.882 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.893 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.894 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.905 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.916 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.917 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.930 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.943 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.954 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.955 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.966 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.977 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:36.977 INFO:teuthology.task.ansible.out: 2020-03-28T06:04:36.991 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.064 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:37 +0000 (0:00:01.626) 0:01:29.867 ******** 2020-03-28T06:04:37.081 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-03-28T06:04:37.083 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.093 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-28T06:04:37.094 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-28T06:04:37.094 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.139 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:37 +0000 (0:00:00.074) 0:01:29.941 ******** 2020-03-28T06:04:37.146 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.150 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.152 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed', u'name': u'cmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.153 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.154 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.156 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.166 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.177 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.189 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.224 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.225 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.227 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.237 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.248 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.259 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.260 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.271 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.287 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.293 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.304 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.316 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.318 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.331 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.344 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.358 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.369 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.380 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.391 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.405 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.423 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.429 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.443 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.451 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.452 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.464 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.475 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.486 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.498 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.509 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi168.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-03-28T06:04:37.520 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.531 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.533 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.543 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.554 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.577 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.578 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.599 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.599 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.601 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.611 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.623 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.635 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.646 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.646 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.664 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.668 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.679 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.692 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.693 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.704 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.715 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.729 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.740 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.741 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.752 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.763 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.775 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.786 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.796 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.819 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.831 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-28T06:04:37.833 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.846 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.876 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.881 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.893 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.907 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.919 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.930 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.942 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.954 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.964 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.977 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.988 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:37.999 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:38.010 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:38.022 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:38.023 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:38.034 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:38.048 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad Uw1JHj0kPIqSd9iOJ68TNg 46badc3b5cd150bfb74daf4300c8329491000521738ba70455631aa4a47e42c0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:38.115 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:38 +0000 (0:00:00.976) 0:01:30.918 ******** 2020-03-28T06:04:38.152 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-28T06:04:38.154 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:38.193 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:38 +0000 (0:00:00.077) 0:01:30.995 ******** 2020-03-28T06:04:38.198 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-28T06:04:38.199 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-03-28T06:04:38.534 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-03-28T06:04:38.593 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-03-28T06:04:38.851 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-03-28T06:04:38.946 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-03-28T06:04:39.186 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-03-28T06:04:39.281 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-03-28T06:04:39.497 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-03-28T06:04:39.623 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-03-28T06:04:39.966 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-03-28T06:04:40.019 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "gmeno", "name": "gmeno", "remove": false, "state": "absent"} 2020-03-28T06:04:40.290 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-03-28T06:04:40.328 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-03-28T06:04:40.599 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-03-28T06:04:40.664 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-03-28T06:04:40.999 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-03-28T06:04:41.208 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "gmeno", "name": "gmeno", "remove": false, "state": "absent"} 2020-03-28T06:04:41.324 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-03-28T06:04:41.544 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-03-28T06:04:41.664 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-03-28T06:04:41.909 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-03-28T06:04:41.999 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-03-28T06:04:42.232 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-03-28T06:04:42.317 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-03-28T06:04:42.553 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-03-28T06:04:42.667 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-03-28T06:04:42.875 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-03-28T06:04:42.990 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-03-28T06:04:43.204 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-03-28T06:04:43.314 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-03-28T06:04:43.536 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-03-28T06:04:43.673 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-03-28T06:04:43.874 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-03-28T06:04:44.003 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-03-28T06:04:44.227 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-03-28T06:04:44.342 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-03-28T06:04:44.550 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-03-28T06:04:44.666 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-03-28T06:04:44.918 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-03-28T06:04:44.994 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-03-28T06:04:45.284 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-03-28T06:04:45.285 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-03-28T06:04:45.558 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-03-28T06:04:45.593 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-03-28T06:04:45.909 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-03-28T06:04:45.931 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-03-28T06:04:46.246 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-03-28T06:04:46.269 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-03-28T06:04:46.586 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-03-28T06:04:46.588 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-03-28T06:04:46.922 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-03-28T06:04:46.946 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-03-28T06:04:47.240 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-03-28T06:04:47.272 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-03-28T06:04:47.565 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-03-28T06:04:47.587 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-03-28T06:04:47.918 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-03-28T06:04:47.919 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-03-28T06:04:48.241 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-03-28T06:04:48.253 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-03-28T06:04:48.581 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-03-28T06:04:48.592 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-03-28T06:04:48.885 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-03-28T06:04:48.907 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-03-28T06:04:49.215 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-03-28T06:04:49.233 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-03-28T06:04:49.535 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-03-28T06:04:49.547 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-03-28T06:04:49.855 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-03-28T06:04:49.909 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-03-28T06:04:50.188 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-03-28T06:04:50.231 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-03-28T06:04:50.512 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-03-28T06:04:50.556 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-03-28T06:04:50.859 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-03-28T06:04:50.870 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-03-28T06:04:51.196 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-03-28T06:04:51.231 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-03-28T06:04:51.529 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-03-28T06:04:51.572 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-03-28T06:04:51.627 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:51 +0000 (0:00:13.431) 0:01:44.427 ******** 2020-03-28T06:04:51.926 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-03-28T06:04:52.254 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-03-28T06:04:52.580 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-03-28T06:04:52.721 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-28T06:04:52.723 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "checksum": "1292e3b894a02d49ff85362f40c4ffb36408b41f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "2c8a02c64a65408cdc5516026a008f96", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375491.65-14394474704202/source", "state": "file", "uid": 0} 2020-03-28T06:04:52.760 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:52 +0000 (0:00:01.135) 0:01:45.563 ******** 2020-03-28T06:04:52.801 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-28T06:04:52.803 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => { "msg": "Host smithi168.front.sepia.ceph.com is running CentOS 7.6 (Core)" } 2020-03-28T06:04:52.842 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:52 +0000 (0:00:00.081) 0:01:45.644 ******** 2020-03-28T06:04:52.880 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-03-28T06:04:52.936 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:52 +0000 (0:00:00.093) 0:01:45.738 ******** 2020-03-28T06:04:52.946 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi156.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-03-28T06:04:52.979 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-28T06:04:52.980 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:53.026 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.090) 0:01:45.828 ******** 2020-03-28T06:04:53.058 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.031) 0:01:45.860 ******** 2020-03-28T06:04:53.060 INFO:teuthology.task.ansible.out: 2020-03-28T06:04:53.476 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-28T06:04:53.476 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.106629", "end": "2020-03-28 06:04:53.445062", "rc": 0, "start": "2020-03-28 06:04:53.338433", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-28T06:04:53.544 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.483) 0:01:46.343 ******** 2020-03-28T06:04:53.591 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:53.638 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.096) 0:01:46.440 ******** 2020-03-28T06:04:53.667 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-28T06:04:53.668 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:53.720 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.082) 0:01:46.523 ******** 2020-03-28T06:04:53.747 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-28T06:04:53.748 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:53.789 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.068) 0:01:46.592 ******** 2020-03-28T06:04:53.836 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-28T06:04:53.838 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:53.872 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.082) 0:01:46.674 ******** 2020-03-28T06:04:53.899 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:53.935 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:53 +0000 (0:00:00.063) 0:01:46.737 ******** 2020-03-28T06:04:53.962 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-28T06:04:53.963 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.002 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.066) 0:01:46.804 ******** 2020-03-28T06:04:54.049 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-28T06:04:54.050 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:54.088 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.085) 0:01:46.890 ******** 2020-03-28T06:04:54.103 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-28T06:04:54.104 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "checksum": "1292e3b894a02d49ff85362f40c4ffb36408b41f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "2c8a02c64a65408cdc5516026a008f96", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375493.05-9580114108533/source", "state": "file", "uid": 0} 2020-03-28T06:04:54.140 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.052) 0:01:46.942 ******** 2020-03-28T06:04:54.149 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.208 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.068) 0:01:47.010 ******** 2020-03-28T06:04:54.214 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-28T06:04:54.217 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => { "msg": "Host smithi156.front.sepia.ceph.com is running CentOS 7.6 (Core)" } 2020-03-28T06:04:54.258 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.049) 0:01:47.060 ******** 2020-03-28T06:04:54.264 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-28T06:04:54.265 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.304 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.046) 0:01:47.106 ******** 2020-03-28T06:04:54.310 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi156.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-03-28T06:04:54.346 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.042) 0:01:47.149 ******** 2020-03-28T06:04:54.354 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-28T06:04:54.356 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:54.392 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.046) 0:01:47.195 ******** 2020-03-28T06:04:54.398 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-28T06:04:54.399 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.437 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.044) 0:01:47.239 ******** 2020-03-28T06:04:54.442 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-28T06:04:54.444 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.483 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.046) 0:01:47.285 ******** 2020-03-28T06:04:54.531 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-28T06:04:54.533 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:54.573 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.084) 0:01:47.370 ******** 2020-03-28T06:04:54.598 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-28T06:04:54.599 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.657 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.086) 0:01:47.457 ******** 2020-03-28T06:04:54.682 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-28T06:04:54.684 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.718 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.063) 0:01:47.521 ******** 2020-03-28T06:04:54.750 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.789 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.071) 0:01:47.592 ******** 2020-03-28T06:04:54.828 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-28T06:04:54.829 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:54.869 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.076) 0:01:47.668 ******** 2020-03-28T06:04:54.902 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.106385", "end": "2020-03-28 06:04:54.856011", "rc": 0, "start": "2020-03-28 06:04:54.749626", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:54.937 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.070) 0:01:47.738 ******** 2020-03-28T06:04:54.970 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:54 +0000 (0:00:00.034) 0:01:47.772 ******** 2020-03-28T06:04:54.975 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-28T06:04:54.976 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.020 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.050) 0:01:47.823 ******** 2020-03-28T06:04:55.028 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-28T06:04:55.029 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.072 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.051) 0:01:47.874 ******** 2020-03-28T06:04:55.077 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-28T06:04:55.078 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.119 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.047) 0:01:47.921 ******** 2020-03-28T06:04:55.127 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.182 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.063) 0:01:47.984 ******** 2020-03-28T06:04:55.192 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-28T06:04:55.192 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.246 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.063) 0:01:48.047 ******** 2020-03-28T06:04:55.251 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.294 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.049) 0:01:48.097 ******** 2020-03-28T06:04:55.300 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.353 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.058) 0:01:48.155 ******** 2020-03-28T06:04:55.361 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.403 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.050) 0:01:48.205 ******** 2020-03-28T06:04:55.408 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-28T06:04:55.410 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.449 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.046) 0:01:48.252 ******** 2020-03-28T06:04:55.457 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.498 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.048) 0:01:48.300 ******** 2020-03-28T06:04:55.505 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-28T06:04:55.507 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.546 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.048) 0:01:48.348 ******** 2020-03-28T06:04:55.553 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.591 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.044) 0:01:48.393 ******** 2020-03-28T06:04:55.595 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-28T06:04:55.596 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:55.597 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:55.640 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.049) 0:01:48.442 ******** 2020-03-28T06:04:55.645 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-28T06:04:55.646 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:55.688 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.048) 0:01:48.490 ******** 2020-03-28T06:04:55.693 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-28T06:04:55.693 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.732 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.044) 0:01:48.534 ******** 2020-03-28T06:04:55.749 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-28T06:04:55.784 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.051) 0:01:48.586 ******** 2020-03-28T06:04:55.790 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.834 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.049) 0:01:48.636 ******** 2020-03-28T06:04:55.839 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-28T06:04:55.841 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:55.886 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.049) 0:01:48.685 ******** 2020-03-28T06:04:55.892 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi156.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:55.934 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:55 +0000 (0:00:00.051) 0:01:48.736 ******** 2020-03-28T06:04:55.973 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-28T06:04:55.975 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.015 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.080) 0:01:48.817 ******** 2020-03-28T06:04:56.054 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-28T06:04:56.055 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:56.093 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.078) 0:01:48.895 ******** 2020-03-28T06:04:56.135 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.167 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.073) 0:01:48.969 ******** 2020-03-28T06:04:56.194 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-28T06:04:56.196 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.229 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.062) 0:01:49.032 ******** 2020-03-28T06:04:56.256 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-28T06:04:56.257 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:04:56.259 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.298 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.067) 0:01:49.099 ******** 2020-03-28T06:04:56.329 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.032) 0:01:49.131 ******** 2020-03-28T06:04:56.334 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-28T06:04:56.336 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.377 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.047) 0:01:49.179 ******** 2020-03-28T06:04:56.402 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-28T06:04:56.404 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:56.439 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.062) 0:01:49.241 ******** 2020-03-28T06:04:56.488 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-28T06:04:56.490 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.537 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.088) 0:01:49.330 ******** 2020-03-28T06:04:56.567 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.619 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.092) 0:01:49.422 ******** 2020-03-28T06:04:56.648 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-28T06:04:56.650 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.686 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.066) 0:01:49.489 ******** 2020-03-28T06:04:56.729 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.766 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.079) 0:01:49.568 ******** 2020-03-28T06:04:56.792 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-28T06:04:56.794 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.831 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.063) 0:01:49.631 ******** 2020-03-28T06:04:56.863 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-28T06:04:56.865 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:56.904 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.073) 0:01:49.705 ******** 2020-03-28T06:04:56.942 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi156.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-28T06:04:56.978 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:56 +0000 (0:00:00.075) 0:01:49.780 ******** 2020-03-28T06:04:57.005 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-28T06:04:57.051 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:57 +0000 (0:00:00.070) 0:01:49.851 ******** 2020-03-28T06:04:57.075 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-28T06:04:57.077 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:04:57.113 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:57 +0000 (0:00:00.063) 0:01:49.915 ******** 2020-03-28T06:04:57.452 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-28T06:04:57.453 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'enabled': u'1', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "md5sum": "5170d95301fca1f1242a0fb0a528bbcf", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375496.37-183181483160504/source", "state": "file", "uid": 0} 2020-03-28T06:04:57.465 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-28T06:04:57.466 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:04:57.521 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:57 +0000 (0:00:00.407) 0:01:50.323 ******** 2020-03-28T06:04:58.601 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-28T06:04:58.601 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'enabled': 0, u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "md5sum": "00fe4de6cbff2433b9d92afb443fe308", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375497.46-41837764926055/source", "state": "file", "uid": 0} 2020-03-28T06:04:58.647 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:58 +0000 (0:00:01.122) 0:01:51.446 ******** 2020-03-28T06:04:58.693 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'enabled': u'1', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "md5sum": "5170d95301fca1f1242a0fb0a528bbcf", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375497.58-137334989652640/source", "state": "file", "uid": 0} 2020-03-28T06:04:59.797 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi168.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2020-03-28T06:04:59.834 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-28T06:04:59.834 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'enabled': 0, u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "md5sum": "00fe4de6cbff2433b9d92afb443fe308", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1585375498.7-208285141187083/source", "state": "file", "uid": 0} 2020-03-28T06:04:59.881 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:04:59 +0000 (0:00:01.237) 0:01:52.683 ******** 2020-03-28T06:05:00.721 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-28T06:05:00.722 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2020-03-28T06:05:00.774 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:00 +0000 (0:00:00.892) 0:01:53.576 ******** 2020-03-28T06:05:00.823 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2020-03-28T06:05:01.763 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2020-03-28T06:05:01.798 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:01 +0000 (0:00:01.023) 0:01:54.600 ******** 2020-03-28T06:05:02.020 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-28T06:05:02.021 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.903227", "end": "2020-03-28 06:05:01.991489", "rc": 0, "start": "2020-03-28 06:05:01.088262", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nCleaning up list of fastest mirrors", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Cleaning up list of fastest mirrors"]} 2020-03-28T06:05:02.062 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.264) 0:01:54.865 ******** 2020-03-28T06:05:02.089 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-28T06:05:02.090 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.129 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.063) 0:01:54.929 ******** 2020-03-28T06:05:02.168 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-28T06:05:02.170 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.206 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.079) 0:01:55.008 ******** 2020-03-28T06:05:02.232 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-28T06:05:02.233 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.279 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.072) 0:01:55.081 ******** 2020-03-28T06:05:02.307 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-28T06:05:02.309 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.346 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.067) 0:01:55.149 ******** 2020-03-28T06:05:02.376 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-28T06:05:02.377 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.412 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.065) 0:01:55.215 ******** 2020-03-28T06:05:02.441 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.478 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.065) 0:01:55.280 ******** 2020-03-28T06:05:02.505 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-28T06:05:02.507 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.545 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.066) 0:01:55.347 ******** 2020-03-28T06:05:02.579 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-28T06:05:02.581 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:02.616 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:02 +0000 (0:00:00.071) 0:01:55.418 ******** 2020-03-28T06:05:03.132 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-28T06:05:03.133 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.988783", "end": "2020-03-28 06:05:03.099238", "rc": 0, "start": "2020-03-28 06:05:02.110455", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nCleaning up list of fastest mirrors", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Cleaning up list of fastest mirrors"]} 2020-03-28T06:05:03.182 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.566) 0:01:55.985 ******** 2020-03-28T06:05:03.214 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.262 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.079) 0:01:56.064 ******** 2020-03-28T06:05:03.293 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.345 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.082) 0:01:56.147 ******** 2020-03-28T06:05:03.377 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.411 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.066) 0:01:56.214 ******** 2020-03-28T06:05:03.439 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-28T06:05:03.441 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.476 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.064) 0:01:56.279 ******** 2020-03-28T06:05:03.514 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-28T06:05:03.520 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.561 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.080) 0:01:56.360 ******** 2020-03-28T06:05:03.584 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-28T06:05:03.585 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.621 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.064) 0:01:56.424 ******** 2020-03-28T06:05:03.659 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.711 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.089) 0:01:56.513 ******** 2020-03-28T06:05:03.741 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-28T06:05:03.741 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:03.777 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:03 +0000 (0:00:00.066) 0:01:56.580 ******** 2020-03-28T06:05:04.539 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-28T06:05:04.541 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.581076", "end": "2020-03-28 06:05:04.500424", "rc": 0, "start": "2020-03-28 06:05:02.919348", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:05:04.579 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:04 +0000 (0:00:00.800) 0:01:57.380 ******** 2020-03-28T06:05:05.436 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2020-03-28T06:05:05.476 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:05 +0000 (0:00:00.898) 0:01:58.279 ******** 2020-03-28T06:05:05.504 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:05.544 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:05 +0000 (0:00:00.067) 0:01:58.346 ******** 2020-03-28T06:05:05.550 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-28T06:05:05.551 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.441005", "end": "2020-03-28 06:05:05.500463", "rc": 0, "start": "2020-03-28 06:05:04.059458", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:05:05.597 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:05 +0000 (0:00:00.047) 0:01:58.393 ******** TASK [common : Update apt cache] *********************************************** 2020-03-28T06:05:05.599 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:05.634 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:05 +0000 (0:00:00.043) 0:01:58.437 ******** 2020-03-28T06:05:05.662 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-28T06:05:05.663 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:05.698 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:05 +0000 (0:00:00.063) 0:01:58.500 ******** 2020-03-28T06:05:05.734 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-28T06:05:05.735 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:05.775 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:05 +0000 (0:00:00.076) 0:01:58.577 ******** 2020-03-28T06:05:06.064 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-28T06:05:06.066 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2020-03-28T06:05:06.111 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:06 +0000 (0:00:00.335) 0:01:58.913 ******** 2020-03-28T06:05:06.138 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-28T06:05:06.140 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:06.199 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:06 +0000 (0:00:00.088) 0:01:59.002 ******** 2020-03-28T06:05:06.236 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-28T06:05:06.238 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:06.275 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:06 +0000 (0:00:00.075) 0:01:59.077 ******** 2020-03-28T06:05:06.301 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-28T06:05:06.303 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:06.342 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:06 +0000 (0:00:00.065) 0:01:59.143 ******** 2020-03-28T06:05:06.367 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-28T06:05:06.369 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:05:06.406 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:06 +0000 (0:00:00.065) 0:01:59.208 ******** 2020-03-28T06:05:06.699 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-28T06:05:06.735 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:06 +0000 (0:00:00.329) 0:01:59.538 ******** 2020-03-28T06:05:07.335 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-28T06:05:07.381 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:05:07 +0000 (0:00:00.644) 0:02:00.182 ******** 2020-03-28T06:12:24.864 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-28T06:12:24.895 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2020-03-28T06:12:24.917 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:24 +0000 (0:07:17.531) 0:09:17.713 ******** 2020-03-28T06:12:25.468 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2020-03-28T06:12:25.505 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:25 +0000 (0:00:00.594) 0:09:18.308 ******** 2020-03-28T06:12:25.878 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-28T06:12:25.879 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-28T06:12:26.486 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-28T06:12:26.785 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-28T06:12:27.359 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-28T06:12:27.635 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-28T06:12:27.680 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:27 +0000 (0:00:02.174) 0:09:20.483 ******** 2020-03-28T06:12:27.981 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-28T06:12:28.019 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:28 +0000 (0:00:00.338) 0:09:20.821 ******** 2020-03-28T06:12:28.247 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi156.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-28T06:12:28.293 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:28 +0000 (0:00:00.273) 0:09:21.095 ******** 2020-03-28T06:12:28.617 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-28T06:12:28.617 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-28T06:12:28.670 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:28 +0000 (0:00:00.377) 0:09:21.472 ******** 2020-03-28T06:12:28.973 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-28T06:12:28.975 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-28T06:12:29.491 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-28T06:12:29.850 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-28T06:12:30.364 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-28T06:12:30.656 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-28T06:12:30.702 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:30 +0000 (0:00:02.031) 0:09:23.504 ******** 2020-03-28T06:12:30.729 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-28T06:12:30.730 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:30.768 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:30 +0000 (0:00:00.066) 0:09:23.571 ******** 2020-03-28T06:12:30.806 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-28T06:12:30.807 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-28T06:12:30.842 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:30 +0000 (0:00:00.073) 0:09:23.644 ******** 2020-03-28T06:12:31.171 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-28T06:12:31.172 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-28T06:12:31.212 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:31 +0000 (0:00:00.370) 0:09:24.015 ******** 2020-03-28T06:12:31.261 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:31.272 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:31.295 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:31.316 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:31.358 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:31 +0000 (0:00:00.141) 0:09:24.156 ******** 2020-03-28T06:12:31.486 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-28T06:12:31.487 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-28T06:12:31.531 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:31 +0000 (0:00:00.176) 0:09:24.333 ******** 2020-03-28T06:12:31.564 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:31.607 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:31 +0000 (0:00:00.076) 0:09:24.409 ******** 2020-03-28T06:12:31.644 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-28T06:12:31.646 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-28T06:12:31.683 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:31 +0000 (0:00:00.076) 0:09:24.485 ******** 2020-03-28T06:12:32.020 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-28T06:12:32.022 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-28T06:12:32.072 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:32 +0000 (0:00:00.389) 0:09:24.874 ******** 2020-03-28T06:12:32.132 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi156.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:32.143 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:32.162 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:32.167 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-28T06:12:32.206 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:32 +0000 (0:00:00.134) 0:09:25.009 ******** 2020-03-28T06:12:40.527 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-28T06:12:40.527 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["nagios-plugins", "2.3.1-3.el7.x86_64 from epel"], ["nagios-plugins-load", "2.3.1-3.el7.x86_64 from epel"]]}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.trouble-free.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: packages.oit.ncsu.edu\n * updates: mirror.datto.com\nPackage nagios-common-4.4.3-1.el7.x86_64 already installed and latest version\nPackage nrpe-3.2.1-8.el7.x86_64 already installed and latest version\nResolving Dependencies\n--> Running transaction check\n---> Package nagios-plugins.x86_64 0:2.3.1-1.el7 will be updated\n---> Package nagios-plugins.x86_64 0:2.3.1-3.el7 will be an update\n---> Package nagios-plugins-load.x86_64 0:2.3.1-1.el7 will be updated\n---> Package nagios-plugins-load.x86_64 0:2.3.1-3.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n nagios-plugins x86_64 2.3.1-3.el7 epel 113 k\n nagios-plugins-load x86_64 2.3.1-3.el7 epel 41 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 154 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 1.4 MB/s | 154 kB 00:00 \nRunning transaction check\nRunning t 2020-03-28T06:12:40.529 INFO:teuthology.task.ansible.out:ransaction test\nTransaction test succeeded\nRunning transaction\n Updating : nagios-plugins-2.3.1-3.el7.x86_64 1/4 \n Updating : nagios-plugins-load-2.3.1-3.el7.x86_64 2/4 \n Cleanup : nagios-plugins-load-2.3.1-1.el7.x86_64 3/4 \n Cleanup : nagios-plugins-2.3.1-1.el7.x86_64 4/4 \n Verifying : nagios-plugins-2.3.1-3.el7.x86_64 1/4 \n Verifying : nagios-plugins-load-2.3.1-3.el7.x86_64 2/4 \n Verifying : nagios-plugins-2.3.1-1.el7.x86_64 3/4 \n Verifying : nagios-plugins-load-2.3.1-1.el7.x86_64 4/4 \n\nUpdated:\n nagios-plugins.x86_64 0:2.3.1-3.el7 nagios-plugins-load.x86_64 0:2.3.1-3.el7 \n\nComplete!\n"]} 2020-03-28T06:12:40.567 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:40 +0000 (0:00:08.360) 0:09:33.369 ******** 2020-03-28T06:12:40.603 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:40.666 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:40 +0000 (0:00:00.095) 0:09:33.465 ******** 2020-03-28T06:12:41.382 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-28T06:12:41.383 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["nagios-plugins", "2.3.1-3.el7.x86_64 from epel"], ["nagios-plugins-load", "2.3.1-3.el7.x86_64 from epel"]]}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.trouble-free.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: packages.oit.ncsu.edu\n * updates: mirror.datto.com\nPackage nagios-common-4.4.3-1.el7.x86_64 already installed and latest version\nPackage nrpe-3.2.1-8.el7.x86_64 already installed and latest version\nResolving Dependencies\n--> Running transaction check\n---> Package nagios-plugins.x86_64 0:2.3.1-1.el7 will be updated\n---> Package nagios-plugins.x86_64 0:2.3.1-3.el7 will be an update\n---> Package nagios-plugins-load.x86_64 0:2.3.1-1.el7 will be updated\n---> Package nagios-plugins-load.x86_64 0:2.3.1-3.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n nagios-plugins x86_64 2.3.1-3.el7 epel 113 k\n nagios-plugins-load x86_64 2.3.1-3.el7 epel 41 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 154 k\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 1.4 MB/s | 154 kB 00:00 \nRunning transaction check\nRunning t 2020-03-28T06:12:41.385 INFO:teuthology.task.ansible.out:ransaction test\nTransaction test succeeded\nRunning transaction\n Updating : nagios-plugins-2.3.1-3.el7.x86_64 1/4 \n Updating : nagios-plugins-load-2.3.1-3.el7.x86_64 2/4 \n Cleanup : nagios-plugins-load-2.3.1-1.el7.x86_64 3/4 \n Cleanup : nagios-plugins-2.3.1-1.el7.x86_64 4/4 \n Verifying : nagios-plugins-2.3.1-3.el7.x86_64 1/4 \n Verifying : nagios-plugins-load-2.3.1-3.el7.x86_64 2/4 \n Verifying : nagios-plugins-2.3.1-1.el7.x86_64 3/4 \n Verifying : nagios-plugins-load-2.3.1-1.el7.x86_64 4/4 \n\nUpdated:\n nagios-plugins.x86_64 0:2.3.1-3.el7 nagios-plugins-load.x86_64 0:2.3.1-3.el7 \n\nComplete!\n"]} 2020-03-28T06:12:41.437 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:41 +0000 (0:00:00.774) 0:09:34.240 ******** 2020-03-28T06:12:41.477 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:41.521 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:41 +0000 (0:00:00.083) 0:09:34.323 ******** 2020-03-28T06:12:41.540 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-28T06:12:41.541 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-28T06:12:41.585 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:41 +0000 (0:00:00.063) 0:09:34.387 ******** 2020-03-28T06:12:42.447 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-28T06:12:42.494 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:42 +0000 (0:00:00.909) 0:09:35.296 ******** 2020-03-28T06:12:42.513 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-28T06:12:42.550 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:42 +0000 (0:00:00.055) 0:09:35.352 ******** 2020-03-28T06:12:42.587 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-28T06:12:42.589 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:42.636 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:42 +0000 (0:00:00.086) 0:09:35.438 ******** 2020-03-28T06:12:42.948 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-28T06:12:42.950 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:12:42.986 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:42 +0000 (0:00:00.350) 0:09:35.789 ******** 2020-03-28T06:12:43.336 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-28T06:12:43.337 INFO:teuthology.task.ansible.out:fatal: [smithi168.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-28T06:12:43.395 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:43 +0000 (0:00:00.406) 0:09:36.195 ******** 2020-03-28T06:12:43.429 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-28T06:12:43.431 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:43.472 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:43 +0000 (0:00:00.079) 0:09:36.274 ******** 2020-03-28T06:12:43.477 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-28T06:12:43.478 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-28T06:12:43.515 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:43 +0000 (0:00:00.043) 0:09:36.318 ******** 2020-03-28T06:12:43.542 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-28T06:12:43.544 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:43.579 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:43 +0000 (0:00:00.063) 0:09:36.381 ******** 2020-03-28T06:12:43.918 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi156.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:12:43.960 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:43 +0000 (0:00:00.380) 0:09:36.762 ******** 2020-03-28T06:12:44.337 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-28T06:12:44.338 INFO:teuthology.task.ansible.out:fatal: [smithi156.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-28T06:12:44.377 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:44 +0000 (0:00:00.414) 0:09:37.177 ******** 2020-03-28T06:12:44.404 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-28T06:12:44.406 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:12:44.444 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:44 +0000 (0:00:00.067) 0:09:37.245 ******** 2020-03-28T06:12:44.448 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-28T06:12:44.449 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-28T06:12:44.489 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:44 +0000 (0:00:00.046) 0:09:37.291 ******** 2020-03-28T06:12:45.368 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-28T06:12:45.374 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2020-03-28 06:02:57 UTC", "ActiveEnterTimestampMonotonic": "19470735", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target nss-lookup.target network.target system.slice systemd-journald.socket basic.target var-run.mount time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-03-28 06:02:57 UTC", "AssertTimestampMonotonic": "19469818", "Before": "xdm.service multi-user.target shutdown.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-03-28 06:02:57 UTC", "ConditionTimestampMonotonic": "19469818", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8969", "ExecMainStartTimestamp": "Sat 2020-03-28 06:02:57 UTC", "ExecMainStartTimestampMonotonic": "19470672", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2020-03-28T06:12:45.375 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Sat 2020-03-28 06:02:57 UTC] ; stop_time=[n/a] ; pid=8969 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-03-28 06:02:57 UTC", "InactiveExitTimestampMonotonic": "19470735", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8969", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2020-03-28T06:12:45.377 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Sat 2020-03-28 06:02:57 UTC", "WatchdogTimestampMonotonic": "19470713", "WatchdogUSec": "0"}} 2020-03-28T06:12:45.412 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:45 +0000 (0:00:00.923) 0:09:38.215 ******** 2020-03-28T06:12:45.418 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-03-28T06:12:45.419 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-28T06:12:45.459 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:45 +0000 (0:00:00.047) 0:09:38.262 ******** 2020-03-28T06:12:45.737 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-28T06:12:45.737 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.004581", "end": "2020-03-28 06:12:45.711333", "rc": 0, "start": "2020-03-28 06:12:45.706752", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-28T06:12:45.774 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:45 +0000 (0:00:00.314) 0:09:38.576 ******** 2020-03-28T06:12:45.884 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-28T06:12:45.886 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2020-03-28 06:02:48 UTC", "ActiveEnterTimestampMonotonic": "19095799", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice nss-lookup.target systemd-journald.socket var-run.mount basic.target time-sync.target network.target local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Sat 2020-03-28 06:02:48 UTC", "AssertTimestampMonotonic": "19095261", "Before": "getty@tty1.service xdm.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2020-03-28 06:02:48 UTC", "ConditionTimestampMonotonic": "19095261", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8968", "ExecMainStartTimestamp": "Sat 2020-03-28 06:02:48 UTC", "ExecMainStartTimestampMonotonic": "19095752", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2020-03-28T06:12:45.887 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Sat 2020-03-28 06:02:48 UTC] ; stop_time=[n/a] ; pid=8968 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2020-03-28 06:02:48 UTC", "InactiveExitTimestampMonotonic": "19095799", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8968", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2020-03-28T06:12:45.889 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Sat 2020-03-28 06:02:48 UTC", "WatchdogTimestampMonotonic": "19095781", "WatchdogUSec": "0"}} 2020-03-28T06:12:45.924 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:45 +0000 (0:00:00.149) 0:09:38.726 ******** 2020-03-28T06:12:46.249 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-28T06:12:46.251 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.003990", "end": "2020-03-28 06:12:46.223311", "rc": 0, "start": "2020-03-28 06:12:46.219321", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-28T06:12:46.291 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:46 +0000 (0:00:00.367) 0:09:39.093 ******** 2020-03-28T06:12:46.296 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-28T06:12:46.298 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2020-03-28T06:12:46.342 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:46 +0000 (0:00:00.051) 0:09:39.144 ******** 2020-03-28T06:12:46.757 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2020-03-28T06:12:46.803 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:46 +0000 (0:00:00.461) 0:09:39.606 ******** 2020-03-28T06:12:48.489 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-28T06:12:48.489 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2020-03-28T06:12:48.533 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:48 +0000 (0:00:01.729) 0:09:41.335 ******** 2020-03-28T06:12:48.968 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2020-03-28T06:12:49.008 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:49 +0000 (0:00:00.474) 0:09:41.810 ******** 2020-03-28T06:12:50.094 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-28T06:12:50.110 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-28T06:12:50.132 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:50 +0000 (0:00:01.124) 0:09:42.934 ******** 2020-03-28T06:12:50.340 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-28T06:12:50.383 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:50 +0000 (0:00:00.251) 0:09:43.185 ******** 2020-03-28T06:12:58.490 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-28T06:12:58.490 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:08.026481", "end": "2020-03-28 06:12:58.455776", "failed_when_result": false, "rc": 0, "start": "2020-03-28 06:12:50.429295", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-28T06:12:58.537 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:58 +0000 (0:00:08.147) 0:09:51.333 ******** 2020-03-28T06:12:58.561 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:07.835319", "end": "2020-03-28 06:12:58.517965", "failed_when_result": false, "rc": 0, "start": "2020-03-28 06:12:50.682646", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-28T06:12:58.605 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:58 +0000 (0:00:00.073) 0:09:51.407 ******** 2020-03-28T06:12:59.466 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-28T06:12:59.468 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-28T06:12:59.505 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:59 +0000 (0:00:00.900) 0:09:52.308 ******** 2020-03-28T06:12:59.528 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-28T06:12:59.577 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:59 +0000 (0:00:00.071) 0:09:52.380 ******** 2020-03-28T06:12:59.857 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-28T06:12:59.859 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018083", "end": "2020-03-28 06:12:59.823713", "rc": 0, "start": "2020-03-28 06:12:59.805630", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2020-03-28T06:12:59.899 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:59 +0000 (0:00:00.320) 0:09:52.700 ******** 2020-03-28T06:12:59.906 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013701", "end": "2020-03-28 06:12:59.864426", "rc": 0, "start": "2020-03-28 06:12:59.850725", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2020-03-28T06:12:59.946 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:12:59 +0000 (0:00:00.047) 0:09:52.748 ******** 2020-03-28T06:13:00.222 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-28T06:13:00.223 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009233", "end": "2020-03-28 06:13:00.190200", "rc": 0, "start": "2020-03-28 06:13:00.180967", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:13:00.275 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:00 +0000 (0:00:00.323) 0:09:53.071 ******** 2020-03-28T06:13:00.279 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.017110", "end": "2020-03-28 06:13:00.245325", "rc": 0, "start": "2020-03-28 06:13:00.228215", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:13:00.333 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:00 +0000 (0:00:00.063) 0:09:53.135 ******** 2020-03-28T06:13:08.466 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-28T06:13:08.467 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:07.862730", "end": "2020-03-28 06:13:08.427411", "rc": 0, "start": "2020-03-28 06:13:00.564681", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:13:08.505 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:08 +0000 (0:00:08.172) 0:10:01.308 ******** 2020-03-28T06:13:08.554 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:07.889522", "end": "2020-03-28 06:13:08.517911", "rc": 0, "start": "2020-03-28 06:13:00.628389", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:13:08.616 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:08 +0000 (0:00:00.108) 0:10:01.417 ******** 2020-03-28T06:13:08.876 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-28T06:13:08.913 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:08 +0000 (0:00:00.298) 0:10:01.715 ******** 2020-03-28T06:13:08.939 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2020-03-28T06:13:08.981 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:08 +0000 (0:00:00.065) 0:10:01.781 ******** 2020-03-28T06:13:08.989 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-28T06:13:09.027 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.047) 0:10:01.829 ******** 2020-03-28T06:13:09.032 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-28T06:13:09.032 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-28T06:13:09.079 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.052) 0:10:01.881 ******** 2020-03-28T06:13:09.085 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-28T06:13:09.086 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2020-03-28T06:13:09.124 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.045) 0:10:01.926 ******** 2020-03-28T06:13:09.134 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "podman-docker", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "python3"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2020-03-28T06:13:09.181 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.051) 0:10:01.978 ******** TASK [testnode : Including distro specific variables.] ************************* ok: [smithi156.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-28T06:13:09.220 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.044) 0:10:02.022 ******** 2020-03-28T06:13:09.226 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-28T06:13:09.227 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-28T06:13:09.284 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.051) 0:10:02.074 ******** TASK [testnode : Including major version specific variables.] ****************** ok: [smithi156.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "podman-docker", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "python3"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2020-03-28T06:13:09.323 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.052) 0:10:02.126 ******** 2020-03-28T06:13:09.385 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-28T06:13:09.385 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-28T06:13:09.431 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.107) 0:10:02.233 ******** 2020-03-28T06:13:09.607 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-28T06:13:09.608 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-28T06:13:09.647 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.216) 0:10:02.450 ******** 2020-03-28T06:13:09.760 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-28T06:13:09.805 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:09 +0000 (0:00:00.157) 0:10:02.607 ******** 2020-03-28T06:13:09.978 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-28T06:13:09.980 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2020-03-28T06:13:10.024 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.219) 0:10:02.826 ******** 2020-03-28T06:13:10.052 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-28T06:13:10.053 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:10.095 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.070) 0:10:02.897 ******** 2020-03-28T06:13:10.111 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-28T06:13:10.113 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2020-03-28T06:13:10.153 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.058) 0:10:02.956 ******** 2020-03-28T06:13:10.159 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-28T06:13:10.160 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:10.196 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.042) 0:10:02.999 ******** 2020-03-28T06:13:10.201 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-28T06:13:10.202 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:10.240 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.043) 0:10:03.043 ******** 2020-03-28T06:13:10.267 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-28T06:13:10.268 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:10.302 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.062) 0:10:03.105 ******** 2020-03-28T06:13:10.537 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-28T06:13:10.538 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-28T06:13:10.576 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.273) 0:10:03.379 ******** 2020-03-28T06:13:10.667 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-28T06:13:10.715 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:10 +0000 (0:00:00.138) 0:10:03.517 ******** 2020-03-28T06:13:11.532 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-28T06:13:11.574 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:11 +0000 (0:00:00.858) 0:10:04.376 ******** 2020-03-28T06:13:11.602 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-28T06:13:11.603 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi168"}, "changed": false} 2020-03-28T06:13:11.641 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:11 +0000 (0:00:00.067) 0:10:04.443 ******** 2020-03-28T06:13:11.742 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-28T06:13:11.744 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-28T06:13:11.781 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:11 +0000 (0:00:00.140) 0:10:04.584 ******** 2020-03-28T06:13:11.815 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-28T06:13:11.817 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi156"}, "changed": false} 2020-03-28T06:13:11.866 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:11 +0000 (0:00:00.084) 0:10:04.668 ******** 2020-03-28T06:13:12.663 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-28T06:13:12.664 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi168.front.sepia.ceph.com", "ansible_hostname": "smithi168", "ansible_nodename": "smithi168"}, "changed": true, "name": "smithi168"} 2020-03-28T06:13:12.706 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:12 +0000 (0:00:00.840) 0:10:05.508 ******** 2020-03-28T06:13:12.713 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi156.front.sepia.ceph.com", "ansible_hostname": "smithi156", "ansible_nodename": "smithi156"}, "changed": true, "name": "smithi156"} 2020-03-28T06:13:12.758 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:12 +0000 (0:00:00.051) 0:10:05.560 ******** 2020-03-28T06:13:13.580 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2020-03-28T06:13:13.618 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:13 +0000 (0:00:00.860) 0:10:06.421 ******** 2020-03-28T06:13:13.659 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2020-03-28T06:13:13.696 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:13 +0000 (0:00:00.078) 0:10:06.499 ******** 2020-03-28T06:13:14.490 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-28T06:13:14.490 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-28T06:13:14.530 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:14 +0000 (0:00:00.833) 0:10:07.333 ******** 2020-03-28T06:13:14.595 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-28T06:13:14.658 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:14 +0000 (0:00:00.127) 0:10:07.460 ******** 2020-03-28T06:13:14.659 INFO:teuthology.task.ansible.out: 2020-03-28T06:13:15.417 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-28T06:13:15.418 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-28T06:13:15.463 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:15 +0000 (0:00:00.805) 0:10:08.266 ******** 2020-03-28T06:13:15.554 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-28T06:13:15.598 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:15 +0000 (0:00:00.134) 0:10:08.400 ******** 2020-03-28T06:13:16.374 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-28T06:13:16.375 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-28T06:13:16.413 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:16 +0000 (0:00:00.815) 0:10:09.216 ******** 2020-03-28T06:13:16.545 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-28T06:13:16.590 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:16 +0000 (0:00:00.176) 0:10:09.393 ******** 2020-03-28T06:13:16.713 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-28T06:13:16.714 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-28T06:13:16.750 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:16 +0000 (0:00:00.160) 0:10:09.553 ******** 2020-03-28T06:13:16.917 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-28T06:13:16.957 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:16 +0000 (0:00:00.206) 0:10:09.759 ******** 2020-03-28T06:13:17.058 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-28T06:13:17.060 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2020-03-28T06:13:17.108 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:17 +0000 (0:00:00.151) 0:10:09.910 ******** 2020-03-28T06:13:17.292 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2020-03-28T06:13:17.338 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:17 +0000 (0:00:00.228) 0:10:10.139 ******** 2020-03-28T06:13:17.480 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-28T06:13:17.481 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-28T06:13:17.531 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:17 +0000 (0:00:00.194) 0:10:10.334 ******** 2020-03-28T06:13:17.698 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-28T06:13:17.746 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:17 +0000 (0:00:00.214) 0:10:10.548 ******** 2020-03-28T06:13:18.451 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-28T06:13:18.453 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-28T06:13:18.488 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:18 +0000 (0:00:00.742) 0:10:11.291 ******** 2020-03-28T06:13:18.707 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-28T06:13:18.764 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:18 +0000 (0:00:00.272) 0:10:11.564 ******** 2020-03-28T06:13:19.462 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-28T06:13:19.499 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:19 +0000 (0:00:00.737) 0:10:12.301 ******** 2020-03-28T06:13:19.693 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-28T06:13:19.730 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:19 +0000 (0:00:00.230) 0:10:12.532 ******** 2020-03-28T06:13:21.874 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-28T06:13:21.875 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.051555", "end": "2020-03-28 06:13:21.831208", "rc": 0, "start": "2020-03-28 06:13:19.779653", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:13:21.914 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:21 +0000 (0:00:02.184) 0:10:14.717 ******** 2020-03-28T06:13:21.942 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-28T06:13:21.943 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:21.999 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:21 +0000 (0:00:00.080) 0:10:14.797 ******** 2020-03-28T06:13:22.024 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-28T06:13:22.063 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:22 +0000 (0:00:00.068) 0:10:14.866 ******** 2020-03-28T06:13:22.188 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-28T06:13:22.190 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.122485", "end": "2020-03-28 06:13:22.149192", "rc": 0, "start": "2020-03-28 06:13:20.026707", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:13:22.239 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:22 +0000 (0:00:00.175) 0:10:15.041 ******** 2020-03-28T06:13:22.277 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-28T06:13:22.279 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:22.319 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:22 +0000 (0:00:00.079) 0:10:15.121 ******** 2020-03-28T06:13:22.346 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-28T06:13:22.393 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:22 +0000 (0:00:00.069) 0:10:15.191 ******** 2020-03-28T06:13:23.019 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-28T06:13:23.020 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2020-03-28T06:13:23.334 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2020-03-28T06:13:23.975 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2020-03-28T06:13:24.016 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:01.627) 0:10:16.818 ******** 2020-03-28T06:13:24.042 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-28T06:13:24.087 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.068) 0:10:16.887 ******** 2020-03-28T06:13:24.115 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:24.151 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.066) 0:10:16.953 ******** 2020-03-28T06:13:24.188 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-28T06:13:24.190 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-28T06:13:24.200 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-28T06:13:24.237 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.086) 0:10:17.039 ******** 2020-03-28T06:13:24.276 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-28T06:13:24.277 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2020-03-28T06:13:24.317 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.079) 0:10:17.119 ******** 2020-03-28T06:13:24.351 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.034) 0:10:17.154 ******** 2020-03-28T06:13:24.365 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-28T06:13:24.399 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.047) 0:10:17.201 ******** 2020-03-28T06:13:24.404 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:13:24.406 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-28T06:13:24.443 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.043) 0:10:17.245 ******** 2020-03-28T06:13:24.452 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-28T06:13:24.454 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:24.489 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.046) 0:10:17.291 ******** 2020-03-28T06:13:24.548 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-28T06:13:24.548 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-28T06:13:24.550 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-28T06:13:24.609 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.108) 0:10:17.400 ******** 2020-03-28T06:13:24.635 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-03-28T06:13:24.636 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:24.678 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.080) 0:10:17.481 ******** 2020-03-28T06:13:24.705 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:13:24.707 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-28T06:13:24.744 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.065) 0:10:17.546 ******** 2020-03-28T06:13:24.911 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2020-03-28T06:13:24.956 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:24 +0000 (0:00:00.212) 0:10:17.758 ******** 2020-03-28T06:13:25.213 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2020-03-28T06:13:25.252 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:25 +0000 (0:00:00.295) 0:10:18.053 ******** 2020-03-28T06:13:25.462 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-28T06:13:25.463 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.192491", "end": "2020-03-28 06:13:25.423866", "rc": 0, "start": "2020-03-28 06:13:25.231375", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.trouble-free.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: packages.oit.ncsu.edu\n * updates: mirror.datto.com\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.trouble-free.net", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: packages.oit.ncsu.edu", " * updates: mirror.datto.com", "No unfinished transactions left."]} 2020-03-28T06:13:25.500 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:25 +0000 (0:00:00.249) 0:10:18.303 ******** 2020-03-28T06:13:25.799 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.237239", "end": "2020-03-28 06:13:25.770142", "rc": 0, "start": "2020-03-28 06:13:25.532903", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.trouble-free.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: packages.oit.ncsu.edu\n * updates: mirror.datto.com\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.trouble-free.net", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: packages.oit.ncsu.edu", " * updates: mirror.datto.com", "No unfinished transactions left."]} 2020-03-28T06:13:25.839 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:25 +0000 (0:00:00.338) 0:10:18.641 ******** 2020-03-28T06:13:25.897 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-28T06:13:25.898 INFO:teuthology.task.ansible.out:fatal: [smithi168.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.056579", "end": "2020-03-28 06:13:25.858368", "msg": "non-zero return code", "rc": 1, "start": "2020-03-28 06:13:25.801789", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-28T06:13:25.939 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:25 +0000 (0:00:00.099) 0:10:18.741 ******** 2020-03-28T06:13:25.965 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-28T06:13:25.966 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:26.006 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:26 +0000 (0:00:00.067) 0:10:18.808 ******** 2020-03-28T06:13:26.193 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-28T06:13:26.194 INFO:teuthology.task.ansible.out:fatal: [smithi156.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.036367", "end": "2020-03-28 06:13:26.168776", "msg": "non-zero return code", "rc": 1, "start": "2020-03-28 06:13:26.132409", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-28T06:13:26.239 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:26 +0000 (0:00:00.233) 0:10:19.042 ******** 2020-03-28T06:13:26.277 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-28T06:13:26.278 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:26.336 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:26 +0000 (0:00:00.096) 0:10:19.139 ******** 2020-03-28T06:13:34.519 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-28T06:13:34.520 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.4.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.4 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 2020-03-28T06:13:34.522 INFO:teuthology.task.ansible.out: @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.4 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.4 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.4 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 2/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 3/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 4/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos 2020-03-28T06:13:34.523 INFO:teuthology.task.ansible.out: \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.4 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 \n\nComplete!\n"]} 2020-03-28T06:13:34.559 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:34 +0000 (0:00:08.223) 0:10:27.362 ******** 2020-03-28T06:13:35.101 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.4.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.4 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 2020-03-28T06:13:35.103 INFO:teuthology.task.ansible.out: @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.4 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.4 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.4 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 2/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 3/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 4/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos 2020-03-28T06:13:35.104 INFO:teuthology.task.ansible.out: \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.4 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 \n\nComplete!\n"]} 2020-03-28T06:13:35.138 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:35 +0000 (0:00:00.578) 0:10:27.941 ******** 2020-03-28T06:13:38.005 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-28T06:13:38.006 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2020-03-28T06:13:38.049 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:38 +0000 (0:00:02.907) 0:10:30.848 ******** 2020-03-28T06:13:38.083 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-28T06:13:38.084 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:38.138 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:38 +0000 (0:00:00.092) 0:10:30.940 ******** 2020-03-28T06:13:38.634 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-28T06:13:38.634 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2020-03-28T06:13:38.679 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:38 +0000 (0:00:00.540) 0:10:31.481 ******** 2020-03-28T06:13:38.720 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:38.759 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:38 +0000 (0:00:00.079) 0:10:31.561 ******** 2020-03-28T06:13:55.563 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-28T06:13:55.565 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-21.el7_7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing py 2020-03-28T06:13:55.566 INFO:teuthology.task.ansible.out:thon-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "podman-docker-1.4.4-4.el7.centos.noarch providing podman-docker is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 2020-03-28T06:13:55.567 INFO:teuthology.task.ansible.out:providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "python3-3.6.8-10.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.trouble-free.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: packages.oit.ncsu.edu\n * updates: mirror.datto.com\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.4 for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.4 for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.4 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n============================== 2020-03-28T06:13:55.569 INFO:teuthology.task.ansible.out:==================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.4 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.4 updates 700 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.4 updates 436 k\n\nTransaction Summary\n================================================================================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 12 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 4/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 6/8 \n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 7/8 \n Installing : mod_fastcgi-2. 2020-03-28T06:13:55.570 INFO:teuthology.task.ansible.out:4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 2/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 3/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 4/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 5/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 6/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.4 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 \n\nComplete!\n"]} 2020-03-28T06:13:55.604 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:55 +0000 (0:00:16.845) 0:10:48.406 ******** 2020-03-28T06:13:56.176 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-21.el7_7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing py 2020-03-28T06:13:56.183 INFO:teuthology.task.ansible.out:thon-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "podman-docker-1.4.4-4.el7.centos.noarch providing podman-docker is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 2020-03-28T06:13:56.184 INFO:teuthology.task.ansible.out:providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "python3-3.6.8-10.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.trouble-free.net\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: packages.oit.ncsu.edu\n * updates: mirror.datto.com\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.4 for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.4 for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.4 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n============================== 2020-03-28T06:13:56.186 INFO:teuthology.task.ansible.out:==================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.4 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.4 updates 700 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.4 updates 436 k\n\nTransaction Summary\n================================================================================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 16 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 4/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 6/8 \n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 7/8 \n Installing : mod_fastcgi-2. 2020-03-28T06:13:56.187 INFO:teuthology.task.ansible.out:4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 2/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 3/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.4.x86_64 4/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.4.x86_64 5/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 6/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.4.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.4 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 2020-03-28T06:13:56.187 INFO:teuthology.task.ansible.out: \n qemu-img.x86_64 10:1.5.3-167.el7_7.4 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.4 \n\nComplete!\n"]} 2020-03-28T06:13:56.234 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:56 +0000 (0:00:00.629) 0:10:49.036 ******** 2020-03-28T06:13:56.344 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-28T06:13:56.345 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2020-03-28T06:13:56.383 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:56 +0000 (0:00:00.147) 0:10:49.184 ******** 2020-03-28T06:13:56.934 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2020-03-28T06:13:56.976 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:56 +0000 (0:00:00.594) 0:10:49.778 ******** 2020-03-28T06:13:56.978 INFO:teuthology.task.ansible.out: 2020-03-28T06:13:57.015 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:57.052 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:57 +0000 (0:00:00.075) 0:10:49.854 ******** 2020-03-28T06:13:57.058 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-28T06:13:57.059 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2020-03-28T06:13:57.102 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:57 +0000 (0:00:00.049) 0:10:49.904 ******** 2020-03-28T06:13:57.492 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-28T06:13:57.493 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service basic.target system.slice dbus.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ipset.service ip6tables.service ebtables.service shutdown.target iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2020-03-28T06:13:57.495 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2020-03-28T06:13:57.496 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-28T06:13:57.531 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:57 +0000 (0:00:00.428) 0:10:50.333 ******** 2020-03-28T06:13:57.654 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-28T06:13:57.655 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2020-03-28T06:13:57.693 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:57 +0000 (0:00:00.161) 0:10:50.495 ******** 2020-03-28T06:13:57.729 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-28T06:13:57.731 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:57.768 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:57 +0000 (0:00:00.075) 0:10:50.570 ******** 2020-03-28T06:13:57.939 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-28T06:13:57.941 INFO:teuthology.task.ansible.out:fatal: [smithi168.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-28T06:13:57.978 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:57 +0000 (0:00:00.210) 0:10:50.781 ******** 2020-03-28T06:13:58.221 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-28T06:13:58.223 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target dbus.service polkit.service system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ip6tables.service iptables.service ipset.service shutdown.target ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2020-03-28T06:13:58.225 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2020-03-28T06:13:58.226 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-28T06:13:58.265 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:58 +0000 (0:00:00.286) 0:10:51.067 ******** 2020-03-28T06:13:58.655 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-28T06:13:58.657 INFO:teuthology.task.ansible.out:fatal: [smithi156.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-28T06:13:58.694 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:58 +0000 (0:00:00.429) 0:10:51.496 ******** 2020-03-28T06:13:58.798 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-28T06:13:58.799 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-28T06:13:58.845 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:58 +0000 (0:00:00.148) 0:10:51.647 ******** 2020-03-28T06:13:58.871 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-28T06:13:58.873 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:58.912 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:58 +0000 (0:00:00.067) 0:10:51.715 ******** 2020-03-28T06:13:58.939 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-28T06:13:58.981 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:58 +0000 (0:00:00.069) 0:10:51.784 ******** 2020-03-28T06:13:59.007 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-28T06:13:59.009 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.049 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.067) 0:10:51.852 ******** 2020-03-28T06:13:59.077 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-28T06:13:59.078 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable SELinux] *********************************************** ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-28T06:13:59.120 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.070) 0:10:51.922 ******** 2020-03-28T06:13:59.171 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.051) 0:10:51.974 ******** 2020-03-28T06:13:59.309 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 28 March 2020 06:13:59 +0000 (0:00:00.061) 0:10:52.035 ******** TASK [testnode : Install apt keys] ********************************************* skipping: [smithi168.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} Saturday 28 March 2020 06:13:59 +0000 (0:00:00.046) 0:10:52.082 ******** TASK [testnode : Remove custom repos] ****************************************** 2020-03-28T06:13:59.323 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.043) 0:10:52.126 ******** 2020-03-28T06:13:59.329 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-28T06:13:59.331 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.372 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.048) 0:10:52.174 ******** 2020-03-28T06:13:59.378 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-28T06:13:59.379 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.419 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.044) 0:10:52.218 ******** 2020-03-28T06:13:59.421 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-28T06:13:59.461 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.045) 0:10:52.264 ******** 2020-03-28T06:13:59.466 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-28T06:13:59.469 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.505 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.044) 0:10:52.308 ******** 2020-03-28T06:13:59.511 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-28T06:13:59.512 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.551 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.045) 0:10:52.353 ******** 2020-03-28T06:13:59.555 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-28T06:13:59.557 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.595 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.044) 0:10:52.397 ******** 2020-03-28T06:13:59.601 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-28T06:13:59.602 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.638 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.043) 0:10:52.441 ******** 2020-03-28T06:13:59.643 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-28T06:13:59.643 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.680 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.041) 0:10:52.482 ******** 2020-03-28T06:13:59.688 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-28T06:13:59.690 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.723 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.043) 0:10:52.526 ******** 2020-03-28T06:13:59.729 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-28T06:13:59.767 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.043) 0:10:52.570 ******** 2020-03-28T06:13:59.773 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-28T06:13:59.774 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.810 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.042) 0:10:52.612 ******** 2020-03-28T06:13:59.817 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-28T06:13:59.818 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.859 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.048) 0:10:52.661 ******** 2020-03-28T06:13:59.865 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-28T06:13:59.866 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.902 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.043) 0:10:52.704 ******** 2020-03-28T06:13:59.910 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.949 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.047) 0:10:52.751 ******** 2020-03-28T06:13:59.955 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-28T06:13:59.957 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:13:59.995 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:13:59 +0000 (0:00:00.045) 0:10:52.797 ******** 2020-03-28T06:14:00.000 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-28T06:14:00.001 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.038 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:52.840 ******** 2020-03-28T06:14:00.045 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-28T06:14:00.047 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.083 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.045) 0:10:52.885 ******** 2020-03-28T06:14:00.088 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-28T06:14:00.090 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.127 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:52.929 ******** 2020-03-28T06:14:00.131 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-28T06:14:00.133 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.173 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.046) 0:10:52.976 ******** 2020-03-28T06:14:00.179 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-28T06:14:00.180 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.217 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.019 ******** 2020-03-28T06:14:00.221 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-28T06:14:00.223 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.260 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.063 ******** 2020-03-28T06:14:00.273 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.304 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.106 ******** 2020-03-28T06:14:00.310 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-28T06:14:00.311 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.347 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.150 ******** 2020-03-28T06:14:00.361 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.393 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.045) 0:10:53.195 ******** 2020-03-28T06:14:00.399 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.437 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.044) 0:10:53.240 ******** 2020-03-28T06:14:00.442 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-28T06:14:00.444 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.482 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.044) 0:10:53.285 ******** 2020-03-28T06:14:00.490 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-28T06:14:00.491 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.526 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.044) 0:10:53.329 ******** 2020-03-28T06:14:00.537 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.570 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.372 ******** 2020-03-28T06:14:00.575 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-28T06:14:00.575 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.617 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.047) 0:10:53.420 ******** 2020-03-28T06:14:00.622 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-28T06:14:00.623 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.661 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.463 ******** 2020-03-28T06:14:00.666 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-28T06:14:00.668 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.706 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.044) 0:10:53.508 ******** 2020-03-28T06:14:00.713 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-28T06:14:00.714 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.751 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.044) 0:10:53.553 ******** 2020-03-28T06:14:00.757 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-28T06:14:00.758 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.795 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.597 ******** 2020-03-28T06:14:00.799 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-28T06:14:00.801 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.838 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.043) 0:10:53.641 ******** 2020-03-28T06:14:00.844 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-28T06:14:00.846 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.884 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.045) 0:10:53.687 ******** 2020-03-28T06:14:00.892 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.929 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.044) 0:10:53.732 ******** 2020-03-28T06:14:00.938 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-28T06:14:00.940 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:00.977 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:00 +0000 (0:00:00.048) 0:10:53.780 ******** 2020-03-28T06:14:00.983 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-28T06:14:00.984 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.033 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.044) 0:10:53.825 ******** TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.068 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.045) 0:10:53.870 ******** 2020-03-28T06:14:01.073 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-28T06:14:01.074 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.114 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.046) 0:10:53.916 ******** 2020-03-28T06:14:01.119 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-28T06:14:01.121 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.158 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.043) 0:10:53.960 ******** 2020-03-28T06:14:01.166 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-28T06:14:01.167 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.210 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.045) 0:10:54.006 ******** TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-28T06:14:01.212 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.255 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.045) 0:10:54.051 ******** TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.295 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.045) 0:10:54.097 ******** 2020-03-28T06:14:01.300 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-28T06:14:01.301 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.338 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.043) 0:10:54.141 ******** 2020-03-28T06:14:01.344 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-28T06:14:01.345 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.382 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.043) 0:10:54.184 ******** 2020-03-28T06:14:01.389 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-28T06:14:01.390 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.426 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.044) 0:10:54.228 ******** 2020-03-28T06:14:01.431 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-28T06:14:01.432 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.470 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.044) 0:10:54.272 ******** 2020-03-28T06:14:01.476 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:14:01.477 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.518 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.047) 0:10:54.320 ******** 2020-03-28T06:14:01.523 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-28T06:14:01.524 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.563 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.045) 0:10:54.365 ******** 2020-03-28T06:14:01.572 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.608 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.045) 0:10:54.410 ******** 2020-03-28T06:14:01.613 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-28T06:14:01.614 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.652 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.043) 0:10:54.454 ******** 2020-03-28T06:14:01.657 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-28T06:14:01.658 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.695 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.043) 0:10:54.498 ******** 2020-03-28T06:14:01.703 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-28T06:14:01.705 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.743 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.047) 0:10:54.545 ******** 2020-03-28T06:14:01.749 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-28T06:14:01.750 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.791 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.048) 0:10:54.593 ******** 2020-03-28T06:14:01.797 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:14:01.799 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.838 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.047) 0:10:54.641 ******** 2020-03-28T06:14:01.849 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-28T06:14:01.850 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.888 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.049) 0:10:54.690 ******** 2020-03-28T06:14:01.894 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-28T06:14:01.895 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:01.931 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:01 +0000 (0:00:00.043) 0:10:54.734 ******** 2020-03-28T06:14:01.974 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-28T06:14:01.976 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:02.010 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:02 +0000 (0:00:00.078) 0:10:54.812 ******** 2020-03-28T06:14:02.037 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-28T06:14:02.039 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:02.075 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:02 +0000 (0:00:00.064) 0:10:54.877 ******** 2020-03-28T06:14:02.102 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-28T06:14:02.103 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:02.139 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:02 +0000 (0:00:00.064) 0:10:54.941 ******** 2020-03-28T06:14:02.210 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-28T06:14:02.211 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-28T06:14:02.258 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:02 +0000 (0:00:00.119) 0:10:55.061 ******** 2020-03-28T06:14:02.476 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-28T06:14:02.521 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:02 +0000 (0:00:00.258) 0:10:55.319 ******** 2020-03-28T06:14:02.598 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-28T06:14:02.600 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-28T06:14:02.640 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:02 +0000 (0:00:00.122) 0:10:55.442 ******** 2020-03-28T06:14:02.871 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-28T06:14:02.914 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:02 +0000 (0:00:00.273) 0:10:55.716 ******** 2020-03-28T06:14:02.965 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-28T06:14:02.966 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. 2020-03-28T06:14:02.967 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-28T06:14:03.011 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.097) 0:10:55.813 ******** 2020-03-28T06:14:03.039 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-28T06:14:03.040 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.092 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.080) 0:10:55.894 ******** 2020-03-28T06:14:03.120 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-28T06:14:03.121 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.171 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.079) 0:10:55.973 ******** 2020-03-28T06:14:03.199 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-28T06:14:03.200 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.240 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.068) 0:10:56.042 ******** 2020-03-28T06:14:03.256 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-28T06:14:03.257 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-28T06:14:03.297 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.057) 0:10:56.099 ******** 2020-03-28T06:14:03.303 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-28T06:14:03.305 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.342 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.045) 0:10:56.145 ******** 2020-03-28T06:14:03.348 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-28T06:14:03.349 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.387 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.044) 0:10:56.190 ******** 2020-03-28T06:14:03.394 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.441 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.044) 0:10:56.234 ******** TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.476 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.044) 0:10:56.278 ******** 2020-03-28T06:14:03.483 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-28T06:14:03.485 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.522 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.046) 0:10:56.324 ******** 2020-03-28T06:14:03.528 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-28T06:14:03.528 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.566 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.044) 0:10:56.369 ******** 2020-03-28T06:14:03.578 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.613 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.046) 0:10:56.415 ******** 2020-03-28T06:14:03.618 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-28T06:14:03.620 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.658 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.044) 0:10:56.460 ******** 2020-03-28T06:14:03.663 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-28T06:14:03.665 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.701 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.043) 0:10:56.504 ******** 2020-03-28T06:14:03.707 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-28T06:14:03.710 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.749 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.047) 0:10:56.552 ******** 2020-03-28T06:14:03.756 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-28T06:14:03.757 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.798 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.044) 0:10:56.596 ******** 2020-03-28T06:14:03.799 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-28T06:14:03.801 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.839 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.044) 0:10:56.641 ******** 2020-03-28T06:14:03.847 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.884 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.045) 0:10:56.687 ******** 2020-03-28T06:14:03.924 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:03.949 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:03 +0000 (0:00:00.064) 0:10:56.751 ******** 2020-03-28T06:14:03.959 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-28T06:14:03.961 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.002 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.052) 0:10:56.804 ******** 2020-03-28T06:14:04.011 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.049 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.047) 0:10:56.852 ******** 2020-03-28T06:14:04.057 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-28T06:14:04.063 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.101 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.051) 0:10:56.904 ******** 2020-03-28T06:14:04.107 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-28T06:14:04.108 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.148 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.046) 0:10:56.951 ******** 2020-03-28T06:14:04.157 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-28T06:14:04.160 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.197 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.048) 0:10:56.999 ******** 2020-03-28T06:14:04.203 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-28T06:14:04.206 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.244 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.046) 0:10:57.046 ******** 2020-03-28T06:14:04.250 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-28T06:14:04.252 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.290 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.045) 0:10:57.092 ******** 2020-03-28T06:14:04.295 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-28T06:14:04.297 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.334 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.044) 0:10:57.136 ******** 2020-03-28T06:14:04.340 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-28T06:14:04.341 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.380 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.044) 0:10:57.181 ******** 2020-03-28T06:14:04.384 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-28T06:14:04.386 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.423 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.044) 0:10:57.226 ******** 2020-03-28T06:14:04.429 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-28T06:14:04.431 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.468 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.045) 0:10:57.271 ******** 2020-03-28T06:14:04.473 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-28T06:14:04.475 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.511 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.042) 0:10:57.314 ******** 2020-03-28T06:14:04.520 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-28T06:14:04.522 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.558 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.046) 0:10:57.360 ******** 2020-03-28T06:14:04.563 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-28T06:14:04.565 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.612 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.054) 0:10:57.414 ******** 2020-03-28T06:14:04.617 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-28T06:14:04.618 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.657 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.045) 0:10:57.459 ******** 2020-03-28T06:14:04.663 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-28T06:14:04.664 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.703 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.046) 0:10:57.506 ******** 2020-03-28T06:14:04.709 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-28T06:14:04.710 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.746 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.042) 0:10:57.549 ******** 2020-03-28T06:14:04.757 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.794 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.046) 0:10:57.595 ******** 2020-03-28T06:14:04.799 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-28T06:14:04.801 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.836 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.042) 0:10:57.638 ******** 2020-03-28T06:14:04.841 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-28T06:14:04.843 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.880 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.043) 0:10:57.682 ******** 2020-03-28T06:14:04.886 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-28T06:14:04.888 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.924 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.043) 0:10:57.726 ******** 2020-03-28T06:14:04.933 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:04.971 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:04 +0000 (0:00:00.047) 0:10:57.773 ******** 2020-03-28T06:14:04.980 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-28T06:14:04.982 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.020 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.049) 0:10:57.823 ******** 2020-03-28T06:14:05.026 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-28T06:14:05.028 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.066 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.046) 0:10:57.869 ******** 2020-03-28T06:14:05.072 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:14:05.073 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-28T06:14:05.113 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.045) 0:10:57.914 ******** 2020-03-28T06:14:05.117 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-28T06:14:05.119 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.156 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.044) 0:10:57.959 ******** 2020-03-28T06:14:05.162 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-28T06:14:05.163 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.185 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-28T06:14:05.186 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.187 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi168.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.229 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.069) 0:10:58.028 ******** 2020-03-28T06:14:05.231 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.234 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.279 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.049) 0:10:58.077 ******** 2020-03-28T06:14:05.281 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.319 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.044) 0:10:58.121 ******** 2020-03-28T06:14:05.356 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:14:05.358 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-28T06:14:05.396 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.076) 0:10:58.198 ******** 2020-03-28T06:14:05.432 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-28T06:14:05.434 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003849", "end": "2020-03-28 06:14:05.403963", "rc": 0, "start": "2020-03-28 06:14:05.400114", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-28T06:14:05.476 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.080) 0:10:58.278 ******** 2020-03-28T06:14:05.483 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-28T06:14:05.484 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi156.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-28T06:14:05.485 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.507 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.507 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.518 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.519 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi156.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.563 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.086) 0:10:58.365 ******** 2020-03-28T06:14:05.568 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.622 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.059) 0:10:58.424 ******** 2020-03-28T06:14:05.628 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-28T06:14:05.629 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.666 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.043) 0:10:58.468 ******** 2020-03-28T06:14:05.703 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-28T06:14:05.704 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.738 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.072) 0:10:58.541 ******** 2020-03-28T06:14:05.765 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-28T06:14:05.765 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003222", "end": "2020-03-28 06:14:05.741904", "rc": 0, "start": "2020-03-28 06:14:05.738682", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-28T06:14:05.769 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.806 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.067) 0:10:58.609 ******** 2020-03-28T06:14:05.840 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.033) 0:10:58.642 ******** 2020-03-28T06:14:05.873 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.909 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.065) 0:10:58.708 ******** 2020-03-28T06:14:05.942 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-28T06:14:05.944 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:05.980 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:05 +0000 (0:00:00.074) 0:10:58.782 ******** 2020-03-28T06:14:06.007 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:06.045 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:06 +0000 (0:00:00.062) 0:10:58.845 ******** 2020-03-28T06:14:06.071 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-28T06:14:06.073 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:06.115 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:06 +0000 (0:00:00.072) 0:10:58.917 ******** 2020-03-28T06:14:06.343 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2020-03-28T06:14:06.381 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:06 +0000 (0:00:00.265) 0:10:59.183 ******** 2020-03-28T06:14:06.431 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-28T06:14:06.433 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={u'block_used': 13058846, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227265638, u'size_available': 930880053248, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60963527, u'device': u'/dev/sda1', u'inode_used': 91449, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227265638, "block_size": 4096, "block_total": 240324484, "block_used": 13058846, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60963527, "inode_total": 61054976, "inode_used": 91449, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930880053248, "size_total": 984369086464, "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}} 2020-03-28T06:14:06.471 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:06 +0000 (0:00:00.090) 0:10:59.273 ******** 2020-03-28T06:14:06.814 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-28T06:14:06.815 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2020-03-28T06:14:06.857 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:06 +0000 (0:00:00.385) 0:10:59.659 ******** 2020-03-28T06:14:06.863 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-28T06:14:06.864 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.013625", "end": "2020-03-28 06:14:06.806273", "rc": 0, "start": "2020-03-28 06:14:06.792648", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:06.911 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:06 +0000 (0:00:00.054) 0:10:59.714 ******** 2020-03-28T06:14:06.917 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi156.front.sepia.ceph.com] => (item={u'block_used': 13058838, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227265646, u'size_available': 930880086016, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60963531, u'device': u'/dev/sda1', u'inode_used': 91445, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227265646, "block_size": 4096, "block_total": 240324484, "block_used": 13058838, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60963531, "inode_total": 61054976, "inode_used": 91445, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930880086016, "size_total": 984369086464, "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}} 2020-03-28T06:14:06.956 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:06 +0000 (0:00:00.045) 0:10:59.759 ******** 2020-03-28T06:14:06.962 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-28T06:14:06.963 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'block_used': 13058846, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227265638, u'size_available': 930880053248, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60963527, u'device': u'/dev/sda1', u'inode_used': 91449, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227265638, "block_size": 4096, "block_total": 240324484, "block_used": 13058846, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60963527, "inode_total": 61054976, "inode_used": 91449, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930880053248, "size_total": 984369086464, "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:07.001 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:07 +0000 (0:00:00.044) 0:10:59.804 ******** 2020-03-28T06:14:07.028 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-28T06:14:07.064 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:07 +0000 (0:00:00.062) 0:10:59.866 ******** 2020-03-28T06:14:07.104 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-28T06:14:07.105 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFGH', u'wwn-0x5000c50092924459'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFGH', u'wwn': u'0x5000c50092924459', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1', u'wwn-0x5000c50092924459-part1'], u'uuids': [u'4de53983-240a-4436-8d72-98c50b3c74e0']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH", "wwn-0x5000c50092924459"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1", "wwn-0x5000c50092924459-part1"], "labels": [], "masters": [], "uuids": ["4de53983-240a-4436-8d72-98c50b3c74e0"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFGH", "size": "931.51 GB", "support_discard": "0", "vendor": " 2020-03-28T06:14:07.106 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50092924459"}}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:07.286 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-28T06:14:07.288 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.014308", "end": "2020-03-28 06:14:07.258682", "rc": 0, "start": "2020-03-28 06:14:07.244374", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:07.328 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:07 +0000 (0:00:00.263) 0:11:00.130 ******** 2020-03-28T06:14:07.367 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-28T06:14:07.367 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'block_used': 13058838, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227265646, u'size_available': 930880086016, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60963531, u'device': u'/dev/sda1', u'inode_used': 91445, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227265646, "block_size": 4096, "block_total": 240324484, "block_used": 13058838, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60963531, "inode_total": 61054976, "inode_used": 91445, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930880086016, "size_total": 984369086464, "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:07.406 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:07 +0000 (0:00:00.078) 0:11:00.209 ******** 2020-03-28T06:14:07.437 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-28T06:14:07.478 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:07 +0000 (0:00:00.069) 0:11:00.278 ******** 2020-03-28T06:14:07.515 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-28T06:14:07.516 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTP4', u'wwn-0x5000c5009293f21d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XTP4', u'wwn': u'0x5000c5009293f21d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTP4-part1', u'wwn-0x5000c5009293f21d-part1'], u'uuids': [u'4de53983-240a-4436-8d72-98c50b3c74e0']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTP4", "wwn-0x5000c5009293f21d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTP4-part1", "wwn-0x5000c5009293f21d-part1"], "labels": [], "masters": [], "uuids": ["4de53983-240a-4436-8d72-98c50b3c74e0"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTP4", "size": "931.51 GB", "support_discard": "0", "vendor": " 2020-03-28T06:14:07.517 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009293f21d"}}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:09.042 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-D6G4TB-hKAO-5RJD-lM7A-bzIr-iejE-KefKam', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN', u'nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.618261", "end": "2020-03-28 06:14:09.008204", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-D6G4TB-hKAO-5RJD-lM7A-bzIr-iejE-KefKam", "nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN", "nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:07.389 2020-03-28T06:14:09.044 INFO:teuthology.task.ansible.out:943", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:09.448 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-9l2WPj-N53m-L0pv-UAPB-Brwi-pTRr-loLZbp', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GU400BGN', u'nvme-nvme.8086-43564654363233333030475534303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.597086", "end": "2020-03-28 06:14:09.406896", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-9l2WPj-N53m-L0pv-UAPB-Brwi-pTRr-loLZbp", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GU400BGN", "nvme-nvme.8086-43564654363233333030475534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:07.809 2020-03-28T06:14:09.449 INFO:teuthology.task.ansible.out:810", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:10.428 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAoenf9R0872BreU6okvkCYi31jyolqAbw'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.062690", "end": "2020-03-28 06:14:10.391166", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAoenf9R0872BreU6okvkCYi31jyolqAbw"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:09.328476", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:10.837 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdBt2DIMAQQl3HcZX9DY2A5b68EO53jMis'], u'uuids': [u'90a0fe29-a833-4705-ab2b-350ae076c7b2']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.056857", "end": "2020-03-28 06:14:10.795994", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdBt2DIMAQQl3HcZX9DY2A5b68EO53jMis"], "labels": [], "masters": [], "uuids": ["90a0fe29-a833-4705-ab2b-350ae076c7b2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:09.739137", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:11.806 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAV2TS2u5kPWQF7HufPH4oSBiTc8pZ1nKR'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.055698", "end": "2020-03-28 06:14:11.767778", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAV2TS2u5kPWQF7HufPH4oSBiTc8pZ1nKR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:10.712080", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:12.181 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdr0GMvOONN5zTdUPk1rTkRwupGeJgff19'], u'uuids': [u'c989aa17-e468-40b7-8a9b-0459fffd70dd']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.049241", "end": "2020-03-28 06:14:12.147945", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdr0GMvOONN5zTdUPk1rTkRwupGeJgff19"], "labels": [], "masters": [], "uuids": ["c989aa17-e468-40b7-8a9b-0459fffd70dd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:11.098704", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:13.157 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAM3rNtFYI2t1dtZeSJ5x4TNyWrXfDPshB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.056723", "end": "2020-03-28 06:14:13.116758", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAM3rNtFYI2t1dtZeSJ5x4TNyWrXfDPshB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:12.060035", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:13.542 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdTXl9617IwC9w3XvGYUZEJo1Hv1L0SveH'], u'uuids': [u'4c37200f-af51-49aa-b994-426d453edb2e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.048311", "end": "2020-03-28 06:14:13.506826", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdTXl9617IwC9w3XvGYUZEJo1Hv1L0SveH"], "labels": [], "masters": [], "uuids": ["4c37200f-af51-49aa-b994-426d453edb2e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:12.458515", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:14.523 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAxRalqI8tP5MFcz50icEzhZI9XTQp67lY'], u'uuids': [u'96e7becc-8ed3-44ea-b2bb-1a3212ea4467']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.088192", "end": "2020-03-28 06:14:14.490639", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAxRalqI8tP5MFcz50icEzhZI9XTQp67lY"], "labels": [], "masters": [], "uuids": ["96e7becc-8ed3-44ea-b2bb-1a3212ea4467"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:13.402447", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:14.923 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdW68cr2xjPeCENBfqpsb9KCKEmYIunkOp'], u'uuids': [u'8142c49e-abee-4ed0-97a8-46d2f4835c4f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.060888", "end": "2020-03-28 06:14:14.883443", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdW68cr2xjPeCENBfqpsb9KCKEmYIunkOp"], "labels": [], "masters": [], "uuids": ["8142c49e-abee-4ed0-97a8-46d2f4835c4f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:13.822555", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:15.867 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAkghvsDIoZDWCHvLm6DLSymLg14jGqV9E'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.053273", "end": "2020-03-28 06:14:15.832804", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAkghvsDIoZDWCHvLm6DLSymLg14jGqV9E"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:14.779531", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:15.925 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:15 +0000 (0:00:08.448) 0:11:08.727 ******** 2020-03-28T06:14:15.975 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-28T06:14:15.976 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFGH', u'wwn-0x5000c50092924459'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFGH', u'wwn': u'0x5000c50092924459', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1', u'wwn-0x5000c50092924459-part1'], u'uuids': [u'4de53983-240a-4436-8d72-98c50b3c74e0']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH", "wwn-0x5000c50092924459"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1", "wwn-0x5000c50092924459-part1"], "labels": [], "masters": [], "uuids": ["4de53983-240a-4436-8d72-98c50b3c74e0"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFGH", "size": "931.51 GB", "support_discard": "0", "vendor": " 2020-03-28T06:14:15.977 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50092924459"}}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:16.286 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-28T06:14:16.287 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQd4MONVNNtrkFwcn41f2X7cGLm3b6WkArE'], u'uuids': [u'0fb21032-edd9-4cf8-affc-7b317349b0a0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.058964", "end": "2020-03-28 06:14:16.248633", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQd4MONVNNtrkFwcn41f2X7cGLm3b6WkArE"], "labels": [], "masters": [], "uuids": ["0fb21032-edd9-4cf8-affc-7b317349b0a0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:15.189669", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-28T06:14:16.339 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:16 +0000 (0:00:00.413) 0:11:09.141 ******** 2020-03-28T06:14:16.345 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-28T06:14:16.346 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-D6G4TB-hKAO-5RJD-lM7A-bzIr-iejE-KefKam', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN', u'nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.061944", "end": "2020-03-28 06:14:16.302510", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-D6G4TB-hKAO-5RJD-lM7A-bzIr-iejE-KefKam", "nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN", "nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:16.240566", "stderr": 2020-03-28T06:14:16.348 INFO:teuthology.task.ansible.out: "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0555069 s, 2.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0555069 s, 2.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:16.384 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTP4', u'wwn-0x5000c5009293f21d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XTP4', u'wwn': u'0x5000c5009293f21d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTP4-part1', u'wwn-0x5000c5009293f21d-part1'], u'uuids': [u'4de53983-240a-4436-8d72-98c50b3c74e0']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTP4", "wwn-0x5000c5009293f21d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTP4-part1", "wwn-0x5000c5009293f21d-part1"], "labels": [], "masters": [], "uuids": ["4de53983-240a-4436-8d72-98c50b3c74e0"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTP4", "size": "931.51 GB", "support_discard": "0", "vendor": " 2020-03-28T06:14:16.386 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c5009293f21d"}}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:16.781 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-9l2WPj-N53m-L0pv-UAPB-Brwi-pTRr-loLZbp', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GU400BGN', u'nvme-nvme.8086-43564654363233333030475534303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.078219", "end": "2020-03-28 06:14:16.728173", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-9l2WPj-N53m-L0pv-UAPB-Brwi-pTRr-loLZbp", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GU400BGN", "nvme-nvme.8086-43564654363233333030475534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:16.649954", "stderr": 2020-03-28T06:14:16.783 INFO:teuthology.task.ansible.out: "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0716174 s, 1.6 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0716174 s, 1.6 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:16.794 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAoenf9R0872BreU6okvkCYi31jyolqAbw'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.195695", "end": "2020-03-28 06:14:16.753948", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAoenf9R0872BreU6okvkCYi31jyolqAbw"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:16.558253", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.189779 s, 608 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.189779 s, 608 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:17.240 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdBt2DIMAQQl3HcZX9DY2A5b68EO53jMis'], u'uuids': [u'90a0fe29-a833-4705-ab2b-350ae076c7b2']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.171097", "end": "2020-03-28 06:14:17.213407", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdBt2DIMAQQl3HcZX9DY2A5b68EO53jMis"], "labels": [], "masters": [], "uuids": ["90a0fe29-a833-4705-ab2b-350ae076c7b2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:17.042310", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.165749 s, 696 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.165749 s, 696 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:17.273 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAV2TS2u5kPWQF7HufPH4oSBiTc8pZ1nKR'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.189586", "end": "2020-03-28 06:14:17.237247", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAV2TS2u5kPWQF7HufPH4oSBiTc8pZ1nKR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:17.047661", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.183362 s, 629 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.183362 s, 629 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:17.768 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdr0GMvOONN5zTdUPk1rTkRwupGeJgff19'], u'uuids': [u'c989aa17-e468-40b7-8a9b-0459fffd70dd']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.198849", "end": "2020-03-28 06:14:17.733593", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdr0GMvOONN5zTdUPk1rTkRwupGeJgff19"], "labels": [], "masters": [], "uuids": ["c989aa17-e468-40b7-8a9b-0459fffd70dd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:17.534744", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.192182 s, 600 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.192182 s, 600 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:17.793 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAM3rNtFYI2t1dtZeSJ5x4TNyWrXfDPshB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.193078", "end": "2020-03-28 06:14:17.735456", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAM3rNtFYI2t1dtZeSJ5x4TNyWrXfDPshB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:17.542378", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.186577 s, 618 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.186577 s, 618 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:18.270 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdTXl9617IwC9w3XvGYUZEJo1Hv1L0SveH'], u'uuids': [u'4c37200f-af51-49aa-b994-426d453edb2e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.180425", "end": "2020-03-28 06:14:18.237810", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdTXl9617IwC9w3XvGYUZEJo1Hv1L0SveH"], "labels": [], "masters": [], "uuids": ["4c37200f-af51-49aa-b994-426d453edb2e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:18.057385", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.174042 s, 663 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.174042 s, 663 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:18.293 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAxRalqI8tP5MFcz50icEzhZI9XTQp67lY'], u'uuids': [u'96e7becc-8ed3-44ea-b2bb-1a3212ea4467']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.203985", "end": "2020-03-28 06:14:18.262027", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAxRalqI8tP5MFcz50icEzhZI9XTQp67lY"], "labels": [], "masters": [], "uuids": ["96e7becc-8ed3-44ea-b2bb-1a3212ea4467"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:18.058042", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.197655 s, 584 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.197655 s, 584 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:18.769 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAkghvsDIoZDWCHvLm6DLSymLg14jGqV9E'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300KZ400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.170038", "end": "2020-03-28 06:14:18.733409", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-uWjdI8W1cfpW6QBREbhspkjNJPbrfVVAkghvsDIoZDWCHvLm6DLSymLg14jGqV9E"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300KZ400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:18.563371", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.163459 s, 706 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.163459 s, 706 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:18.812 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:18 +0000 (0:00:02.472) 0:11:11.614 ******** 2020-03-28T06:14:18.819 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdW68cr2xjPeCENBfqpsb9KCKEmYIunkOp'], u'uuids': [u'8142c49e-abee-4ed0-97a8-46d2f4835c4f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.201435", "end": "2020-03-28 06:14:18.753540", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQdW68cr2xjPeCENBfqpsb9KCKEmYIunkOp"], "labels": [], "masters": [], "uuids": ["8142c49e-abee-4ed0-97a8-46d2f4835c4f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:18.552105", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.195397 s, 590 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.195397 s, 590 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:19.211 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-28T06:14:19.213 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.078297", "end": "2020-03-28 06:14:19.170908", "rc": 0, "start": "2020-03-28 06:14:19.092611", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:19.248 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.436) 0:11:12.051 ******** 2020-03-28T06:14:19.307 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-28T06:14:19.308 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQd4MONVNNtrkFwcn41f2X7cGLm3b6WkArE'], u'uuids': [u'0fb21032-edd9-4cf8-affc-7b317349b0a0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT623300GU400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.207522", "end": "2020-03-28 06:14:19.268251", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MagyBbJdXnyWh0Xp8usSXZfbCKTchcQd4MONVNNtrkFwcn41f2X7cGLm3b6WkArE"], "labels": [], "masters": [], "uuids": ["0fb21032-edd9-4cf8-affc-7b317349b0a0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT623300GU400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-28 06:14:19.060729", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.200865 s, 574 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.200865 s, 574 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:19.354 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.106) 0:11:12.157 ******** 2020-03-28T06:14:19.613 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-28T06:14:19.615 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048856", "end": "2020-03-28 06:14:19.583497", "rc": 0, "start": "2020-03-28 06:14:19.534641", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:19.651 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.296) 0:11:12.453 ******** 2020-03-28T06:14:19.677 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-28T06:14:19.724 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.072) 0:11:12.526 ******** 2020-03-28T06:14:19.752 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-28T06:14:19.754 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:19.795 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.071) 0:11:12.598 ******** 2020-03-28T06:14:19.801 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-28T06:14:19.801 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.086002", "end": "2020-03-28 06:14:19.735372", "rc": 0, "start": "2020-03-28 06:14:19.649370", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:19.841 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.045) 0:11:12.644 ******** 2020-03-28T06:14:19.847 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-28T06:14:19.849 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:19.850 INFO:teuthology.task.ansible.out: 2020-03-28T06:14:19.888 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.046) 0:11:12.690 ******** 2020-03-28T06:14:19.915 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:19.962 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:19 +0000 (0:00:00.073) 0:11:12.764 ******** 2020-03-28T06:14:20.000 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-28T06:14:20.001 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.038 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.076) 0:11:12.841 ******** 2020-03-28T06:14:20.229 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-28T06:14:20.230 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.059395", "end": "2020-03-28 06:14:20.193972", "rc": 0, "start": "2020-03-28 06:14:20.134577", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:20.268 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.229) 0:11:13.070 ******** 2020-03-28T06:14:20.290 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-28T06:14:20.343 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.074) 0:11:13.145 ******** 2020-03-28T06:14:20.348 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-28T06:14:20.349 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:14:20.389 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.046) 0:11:13.191 ******** 2020-03-28T06:14:20.395 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-28T06:14:20.396 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.437 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.048) 0:11:13.240 ******** 2020-03-28T06:14:20.444 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-28T06:14:20.444 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={u'block_used': 13058846, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227265638, u'size_available': 930880053248, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60963527, u'device': u'/dev/sda1', u'inode_used': 91449, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227265638, "block_size": 4096, "block_total": 240324484, "block_used": 13058846, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60963527, "inode_total": 61054976, "inode_used": 91449, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930880053248, "size_total": 984369086464, "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.483 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.045) 0:11:13.286 ******** 2020-03-28T06:14:20.489 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-28T06:14:20.489 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.528 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.044) 0:11:13.330 ******** 2020-03-28T06:14:20.535 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-28T06:14:20.536 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.574 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.046) 0:11:13.377 ******** 2020-03-28T06:14:20.580 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-28T06:14:20.581 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.624 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.049) 0:11:13.426 ******** 2020-03-28T06:14:20.632 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.672 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.048) 0:11:13.474 ******** 2020-03-28T06:14:20.679 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-28T06:14:20.680 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.722 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.050) 0:11:13.524 ******** 2020-03-28T06:14:20.729 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:14:20.730 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.772 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.049) 0:11:13.574 ******** 2020-03-28T06:14:20.825 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.859 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.084) 0:11:13.659 ******** 2020-03-28T06:14:20.883 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.920 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.062) 0:11:13.722 ******** 2020-03-28T06:14:20.945 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:20.981 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:20 +0000 (0:00:00.061) 0:11:13.783 ******** 2020-03-28T06:14:21.049 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi156.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-28T06:14:21.080 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.098) 0:11:13.882 ******** 2020-03-28T06:14:21.139 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-28T06:14:21.140 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={u'block_used': 13058838, u'uuid': u'4de53983-240a-4436-8d72-98c50b3c74e0', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227265646, u'size_available': 930880086016, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60963531, u'device': u'/dev/sda1', u'inode_used': 91445, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227265646, "block_size": 4096, "block_total": 240324484, "block_used": 13058838, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60963531, "inode_total": 61054976, "inode_used": 91445, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930880086016, "size_total": 984369086464, "uuid": "4de53983-240a-4436-8d72-98c50b3c74e0"}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:21.179 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.098) 0:11:13.981 ******** 2020-03-28T06:14:21.220 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:21.263 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.084) 0:11:14.065 ******** 2020-03-28T06:14:21.297 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:21.333 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.070) 0:11:14.136 ******** 2020-03-28T06:14:21.363 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:21.402 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.068) 0:11:14.204 ******** 2020-03-28T06:14:21.439 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-28T06:14:21.440 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:21.478 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.076) 0:11:14.280 ******** 2020-03-28T06:14:21.516 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-28T06:14:21.517 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:21.556 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.078) 0:11:14.359 ******** 2020-03-28T06:14:21.585 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-28T06:14:21.586 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:21.630 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.073) 0:11:14.433 ******** 2020-03-28T06:14:21.847 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-28T06:14:21.848 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-28T06:14:21.888 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:21 +0000 (0:00:00.257) 0:11:14.690 ******** 2020-03-28T06:14:22.157 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-28T06:14:22.195 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:22 +0000 (0:00:00.306) 0:11:14.997 ******** 2020-03-28T06:14:22.657 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:22.712 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:23.153 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:23.204 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:23.604 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:23.689 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:24.065 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:24.139 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:24.519 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:24.558 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:24 +0000 (0:00:02.362) 0:11:17.360 ******** 2020-03-28T06:14:24.629 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-28T06:14:24.667 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:24 +0000 (0:00:00.109) 0:11:17.470 ******** 2020-03-28T06:14:24.863 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-28T06:14:24.864 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-28T06:14:24.900 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:24 +0000 (0:00:00.232) 0:11:17.703 ******** 2020-03-28T06:14:24.940 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-28T06:14:24.941 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:24.989 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-28T06:14:24.991 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-28T06:14:25.028 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:25 +0000 (0:00:00.127) 0:11:17.830 ******** 2020-03-28T06:14:25.066 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-28T06:14:25.066 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:25.258 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:25.379 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:25.568 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:25.695 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:25.878 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:26.008 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:26.183 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:26.227 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:26 +0000 (0:00:01.198) 0:11:19.029 ******** 2020-03-28T06:14:26.328 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-28T06:14:26.377 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:26 +0000 (0:00:00.149) 0:11:19.179 ******** 2020-03-28T06:14:26.546 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-28T06:14:26.547 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-28T06:14:26.585 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:26 +0000 (0:00:00.208) 0:11:19.388 ******** 2020-03-28T06:14:26.622 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-28T06:14:26.624 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-28T06:14:26.662 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:26 +0000 (0:00:00.077) 0:11:19.465 ******** 2020-03-28T06:14:26.699 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-28T06:14:26.701 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-28T06:14:26.737 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:26 +0000 (0:00:00.074) 0:11:19.540 ******** 2020-03-28T06:14:26.789 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-28T06:14:26.791 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-28T06:14:26.829 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:26 +0000 (0:00:00.091) 0:11:19.631 ******** 2020-03-28T06:14:27.364 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-28T06:14:27.366 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true} 2020-03-28T06:14:27.404 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:27 +0000 (0:00:00.574) 0:11:20.206 ******** 2020-03-28T06:14:27.411 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true} 2020-03-28T06:14:27.448 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:27 +0000 (0:00:00.044) 0:11:20.250 ******** 2020-03-28T06:14:28.122 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-28T06:14:28.124 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-28T06:14:28.171 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.722) 0:11:20.973 ******** 2020-03-28T06:14:28.178 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-28T06:14:28.218 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.047) 0:11:21.021 ******** 2020-03-28T06:14:28.225 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-28T06:14:28.226 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.267 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.048) 0:11:21.069 ******** 2020-03-28T06:14:28.276 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.316 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.048) 0:11:21.118 ******** 2020-03-28T06:14:28.322 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-28T06:14:28.324 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.363 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.047) 0:11:21.165 ******** 2020-03-28T06:14:28.369 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.416 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.053) 0:11:21.218 ******** 2020-03-28T06:14:28.425 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.463 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.047) 0:11:21.266 ******** 2020-03-28T06:14:28.470 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.513 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.049) 0:11:21.315 ******** 2020-03-28T06:14:28.522 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-28T06:14:28.524 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.561 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.047) 0:11:21.363 ******** 2020-03-28T06:14:28.567 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.607 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.045) 0:11:21.409 ******** 2020-03-28T06:14:28.613 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-28T06:14:28.615 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.653 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.046) 0:11:21.456 ******** 2020-03-28T06:14:28.659 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.702 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.048) 0:11:21.504 ******** 2020-03-28T06:14:28.708 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-28T06:14:28.710 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.746 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.044) 0:11:21.549 ******** 2020-03-28T06:14:28.755 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.795 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.045) 0:11:21.594 ******** 2020-03-28T06:14:28.798 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-28T06:14:28.798 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.838 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.045) 0:11:21.640 ******** 2020-03-28T06:14:28.844 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:28.885 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:28 +0000 (0:00:00.047) 0:11:21.688 ******** 2020-03-28T06:14:29.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-28T06:14:29.307 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2020-03-28T06:14:29.348 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:29 +0000 (0:00:00.462) 0:11:22.151 ******** 2020-03-28T06:14:29.393 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install ntp package on rpm based systems.] ******************** ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2020-03-28T06:14:29.436 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:29 +0000 (0:00:00.087) 0:11:22.239 ******** 2020-03-28T06:14:29.472 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:29 +0000 (0:00:00.035) 0:11:22.274 ******** 2020-03-28T06:14:29.476 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-28T06:14:29.478 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:29.519 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:29 +0000 (0:00:00.047) 0:11:22.321 ******** 2020-03-28T06:14:29.526 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-28T06:14:29.527 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:29.569 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:29 +0000 (0:00:00.049) 0:11:22.371 ******** 2020-03-28T06:14:29.576 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:29.621 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:29 +0000 (0:00:00.052) 0:11:22.423 ******** 2020-03-28T06:14:30.438 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-28T06:14:30.439 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2020-03-28T06:14:30.482 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:30 +0000 (0:00:00.860) 0:11:23.284 ******** 2020-03-28T06:14:30.518 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-28T06:14:30.519 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:30.556 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:30 +0000 (0:00:00.074) 0:11:23.358 ******** 2020-03-28T06:14:30.560 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-28T06:14:30.561 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2020-03-28T06:14:30.599 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:30 +0000 (0:00:00.042) 0:11:23.401 ******** 2020-03-28T06:14:30.624 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-28T06:14:30.626 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:30.660 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:30 +0000 (0:00:00.061) 0:11:23.463 ******** 2020-03-28T06:14:31.008 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-28T06:14:31.010 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "-.mount syslog.target tmp.mount systemd-journald.socket basic.target sntp.service ntpdate.service system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target multi-user.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2020-03-28T06:14:31.012 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2020-03-28T06:14:31.013 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-28T06:14:31.047 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:31 +0000 (0:00:00.386) 0:11:23.850 ******** 2020-03-28T06:14:31.106 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sntp.service systemd-journald.socket ntpdate.service syslog.target tmp.mount -.mount system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "multi-user.target chronyd.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2020-03-28T06:14:31.107 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2020-03-28T06:14:31.109 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-28T06:14:31.142 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:31 +0000 (0:00:00.094) 0:11:23.944 ******** 2020-03-28T06:14:31.957 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-28T06:14:31.958 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2020-03-28T06:14:31.995 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:31 +0000 (0:00:00.853) 0:11:24.798 ******** 2020-03-28T06:14:32.022 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-28T06:14:32.023 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:32.060 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:32 +0000 (0:00:00.064) 0:11:24.862 ******** 2020-03-28T06:14:32.064 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-28T06:14:32.065 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2020-03-28T06:14:32.105 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:32 +0000 (0:00:00.045) 0:11:24.908 ******** 2020-03-28T06:14:32.111 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-28T06:14:32.113 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:32.149 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:32 +0000 (0:00:00.044) 0:11:24.952 ******** 2020-03-28T06:14:32.155 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-28T06:14:32.164 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:32.204 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:32 +0000 (0:00:00.051) 0:11:25.004 ******** 2020-03-28T06:14:32.228 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-28T06:14:32.229 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:32.267 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:32 +0000 (0:00:00.064) 0:11:25.068 ******** 2020-03-28T06:14:32.481 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-28T06:14:32.481 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-28T06:14:32.520 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:32 +0000 (0:00:00.254) 0:11:25.322 ******** 2020-03-28T06:14:32.579 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-28T06:14:32.618 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:32 +0000 (0:00:00.097) 0:11:25.420 ******** 2020-03-28T06:14:33.450 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-28T06:14:33.452 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-28T06:14:33.491 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:33 +0000 (0:00:00.872) 0:11:26.293 ******** 2020-03-28T06:14:33.510 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-28T06:14:33.555 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:33 +0000 (0:00:00.063) 0:11:26.357 ******** 2020-03-28T06:14:35.263 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 20.0.2 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 20.0.2 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-28T06:14:35.305 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:01.743) 0:11:28.100 ******** 2020-03-28T06:14:35.317 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 20.0.2 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 20.0.2 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-28T06:14:35.318 INFO:teuthology.task.ansible.out: 2020-03-28T06:14:35.361 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.061) 0:11:28.162 ******** 2020-03-28T06:14:35.367 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:35.410 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.049) 0:11:28.212 ******** 2020-03-28T06:14:35.416 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:35.458 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.048) 0:11:28.261 ******** 2020-03-28T06:14:35.463 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-28T06:14:35.464 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-28T06:14:35.509 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.050) 0:11:28.311 ******** 2020-03-28T06:14:35.515 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-28T06:14:35.552 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.043) 0:11:28.355 ******** 2020-03-28T06:14:35.832 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-28T06:14:35.833 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-28T06:14:35.873 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.320) 0:11:28.675 ******** 2020-03-28T06:14:35.880 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-28T06:14:35.920 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.047) 0:11:28.723 ******** 2020-03-28T06:14:35.925 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-28T06:14:35.926 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:35.968 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:35 +0000 (0:00:00.047) 0:11:28.771 ******** 2020-03-28T06:14:35.975 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:36.014 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.046) 0:11:28.817 ******** 2020-03-28T06:14:36.020 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-28T06:14:36.020 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-28T06:14:36.061 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.045) 0:11:28.863 ******** 2020-03-28T06:14:36.068 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-28T06:14:36.104 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.043) 0:11:28.906 ******** 2020-03-28T06:14:36.110 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-28T06:14:36.110 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:36.149 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.045) 0:11:28.952 ******** 2020-03-28T06:14:36.155 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:36.193 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.043) 0:11:28.995 ******** 2020-03-28T06:14:36.469 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-28T06:14:36.471 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-28T06:14:36.512 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.319) 0:11:29.315 ******** 2020-03-28T06:14:36.519 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-28T06:14:36.558 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.045) 0:11:29.360 ******** 2020-03-28T06:14:36.804 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi156.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-28T06:14:36.842 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.284) 0:11:29.645 ******** 2020-03-28T06:14:36.860 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-28T06:14:36.898 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.055) 0:11:29.701 ******** 2020-03-28T06:14:36.904 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-28T06:14:36.906 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:36.952 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:36 +0000 (0:00:00.046) 0:11:29.747 ******** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.016 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.071) 0:11:29.818 ******** 2020-03-28T06:14:37.023 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi156.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.024 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.025 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.069 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.053) 0:11:29.871 ******** 2020-03-28T06:14:37.077 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.079 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.080 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.081 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.135 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.065) 0:11:29.937 ******** 2020-03-28T06:14:37.144 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi156.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi156.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.145 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.146 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.148 INFO:teuthology.task.ansible.out: 2020-03-28T06:14:37.197 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.061) 0:11:29.999 ******** 2020-03-28T06:14:37.204 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.206 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.208 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.209 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.211 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.249 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.051) 0:11:30.051 ******** 2020-03-28T06:14:37.254 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-28T06:14:37.255 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.299 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.050) 0:11:30.101 ******** 2020-03-28T06:14:37.305 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.349 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.050) 0:11:30.152 ******** 2020-03-28T06:14:37.355 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-28T06:14:37.356 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.394 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.045) 0:11:30.197 ******** 2020-03-28T06:14:37.400 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.440 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.045) 0:11:30.243 ******** 2020-03-28T06:14:37.446 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-28T06:14:37.447 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.490 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.048) 0:11:30.291 ******** 2020-03-28T06:14:37.495 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.537 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.047) 0:11:30.339 ******** 2020-03-28T06:14:37.542 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-28T06:14:37.544 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.581 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.044) 0:11:30.383 ******** 2020-03-28T06:14:37.587 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.626 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.044) 0:11:30.428 ******** 2020-03-28T06:14:37.632 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-28T06:14:37.633 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.673 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.046) 0:11:30.475 ******** 2020-03-28T06:14:37.678 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.721 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.047) 0:11:30.523 ******** 2020-03-28T06:14:37.726 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-28T06:14:37.728 INFO:teuthology.task.ansible.out:skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.767 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.046) 0:11:30.569 ******** 2020-03-28T06:14:37.773 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.812 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.045) 0:11:30.615 ******** 2020-03-28T06:14:37.821 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.859 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.046) 0:11:30.661 ******** 2020-03-28T06:14:37.864 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.904 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.045) 0:11:30.706 ******** 2020-03-28T06:14:37.909 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi156.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.948 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.043) 0:11:30.750 ******** 2020-03-28T06:14:37.953 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-28T06:14:37.991 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:37 +0000 (0:00:00.043) 0:11:30.793 ******** 2020-03-28T06:14:38.295 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-28T06:14:38.297 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.013697", "end": "2020-03-28 06:14:38.253901", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-28 06:14:38.240204", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:38.317 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.015277", "end": "2020-03-28 06:14:38.275387", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-28 06:14:38.260110", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:38.622 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.007441", "end": "2020-03-28 06:14:38.584526", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-28 06:14:38.577085", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:38.923 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.006838", "end": "2020-03-28 06:14:38.882293", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-28 06:14:38.875455", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:39.241 INFO:teuthology.task.ansible.out:ok: [smithi156.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.018792", "end": "2020-03-28 06:14:39.210627", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-28 06:14:39.191835", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:39.290 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:39 +0000 (0:00:01.299) 0:11:32.093 ******** 2020-03-28T06:14:39.318 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-28T06:14:39.361 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:39 +0000 (0:00:00.067) 0:11:32.160 ******** 2020-03-28T06:14:39.660 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi168.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:01.008862", "end": "2020-03-28 06:14:39.612238", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-28 06:14:38.603376", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:39.689 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-28T06:14:39.691 INFO:teuthology.task.ansible.out:changed: [smithi156.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-28T06:14:40.004 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-28T06:14:40.005 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.007456", "end": "2020-03-28 06:14:39.953785", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-28 06:14:39.946329", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:40.340 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026674", "end": "2020-03-28 06:14:40.296227", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-28 06:14:40.269553", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-28T06:14:40.387 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:40 +0000 (0:00:01.028) 0:11:33.189 ******** 2020-03-28T06:14:40.414 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-28T06:14:40.460 INFO:teuthology.task.ansible.out:Saturday 28 March 2020 06:14:40 +0000 (0:00:00.073) 0:11:33.263 ******** 2020-03-28T06:14:40.762 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-28T06:14:40.763 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-28T06:14:40.896 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-03-28T06:14:40.899 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-03-28T06:14:40.900 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-28T06:14:40.902 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-03-28T06:14:40.904 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi156.front.sepia.ceph.com : ok=119 changed=40 unreachable=0 failed=0 skipped=164 rescued=0 ignored=3 smithi168.front.sepia.ceph.com : ok=119 changed=40 unreachable=0 failed=0 skipped=164 rescued=0 ignored=3 Saturday 28 March 2020 06:14:40 +0000 (0:00:00.442) 0:11:33.705 ******** =============================================================================== 2020-03-28T06:14:40.905 INFO:teuthology.task.ansible.out:common : Make sure smartmontools is installed ------------------------- 437.54s 2020-03-28T06:14:40.907 INFO:teuthology.task.ansible.out:users : Create all admin users with sudo access. ----------------------- 38.33s users : Update authorized_keys using the keys repo --------------------- 37.29s testnode : Install packages -------------------------------------------- 16.85s users : Remove revoked users ------------------------------------------- 13.43s testnode : Zap all non-root disks --------------------------------------- 8.45s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 8.36s testnode : Ensure ceph packages are not present. ------------------------ 8.22s common : nrpe - Load SELinux policy package ----------------------------- 8.17s common : nrpe - Remove SELinux policy package --------------------------- 8.15s Gathering Facts --------------------------------------------------------- 2.97s testnode : Ensure ceph dependency packages are not present. ------------- 2.91s testnode : Blow away lingering OSD data and FSIDs ----------------------- 2.47s testnode : Create logical volume(s) ------------------------------------- 2.36s testnode : Rebuild rpmdb ------------------------------------------------ 2.18s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 2.17s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 2.03s testnode : Install packages via pip ------------------------------------- 1.74s common : nrpe - Ensure SELinux policy is up to date --------------------- 1.73s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.63s 2020-03-28T06:14:41.192 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi168.front.sepia.ceph.com')] 2020-03-28T06:14:41.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:14:41.195 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:14:41.302 INFO:teuthology.orchestra.run.smithi156:> true 2020-03-28T06:14:41.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T06:14:41.700 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T06:14:41.805 INFO:teuthology.orchestra.run.smithi168:> true 2020-03-28T06:14:42.162 INFO:teuthology.run_tasks:Running task clock... 2020-03-28T06:14:42.174 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-28T06:14:42.174 INFO:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-28T06:14:42.317 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-28T06:14:52.409 INFO:teuthology.orchestra.run.smithi156.stdout:ntpd: time slew -0.002352s 2020-03-28T06:14:52.450 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit not found. 2020-03-28T06:14:52.530 INFO:teuthology.orchestra.run.smithi156.stdout: remote refid st t when poll reach delay offset jitter 2020-03-28T06:14:52.530 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================== 2020-03-28T06:14:52.531 INFO:teuthology.orchestra.run.smithi156.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:14:52.531 INFO:teuthology.orchestra.run.smithi156.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:14:52.531 INFO:teuthology.orchestra.run.smithi156.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:14:52.531 INFO:teuthology.orchestra.run.smithi156.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:14:52.531 INFO:teuthology.orchestra.run.smithi168:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-28T06:14:52.671 INFO:teuthology.orchestra.run.smithi168.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-28T06:15:02.749 INFO:teuthology.orchestra.run.smithi168.stdout:ntpd: time slew -0.002675s 2020-03-28T06:15:02.787 INFO:teuthology.orchestra.run.smithi168.stderr:Failed to start ntp.service: Unit not found. 2020-03-28T06:15:02.866 INFO:teuthology.orchestra.run.smithi168.stdout: remote refid st t when poll reach delay offset jitter 2020-03-28T06:15:02.866 INFO:teuthology.orchestra.run.smithi168.stdout:============================================================================== 2020-03-28T06:15:02.866 INFO:teuthology.orchestra.run.smithi168.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:15:02.866 INFO:teuthology.orchestra.run.smithi168.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:15:02.866 INFO:teuthology.orchestra.run.smithi168.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:15:02.867 INFO:teuthology.orchestra.run.smithi168.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-03-28T06:15:02.867 INFO:teuthology.run_tasks:Running task install... 2020-03-28T06:15:02.878 DEBUG:teuthology.task.install:project ceph 2020-03-28T06:15:02.878 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'notcmalloc', 'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e'}} 2020-03-28T06:15:02.878 DEBUG:teuthology.task.install:config {'flavor': 'notcmalloc', 'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e'} 2020-03-28T06:15:02.878 INFO:teuthology.task.install:Using flavor: notcmalloc 2020-03-28T06:15:02.883 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2020-03-28T06:15:02.883 INFO:teuthology.task.install:extra packages: [] 2020-03-28T06:15:02.883 INFO:teuthology.orchestra.run.smithi156:> sudo lsb_release -is 2020-03-28T06:15:03.056 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS 2020-03-28T06:15:03.057 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-28T06:15:03.057 INFO:teuthology.orchestra.run.smithi168:> sudo lsb_release -is 2020-03-28T06:15:03.057 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'notcmalloc', 'extras': None} 2020-03-28T06:15:03.058 INFO:teuthology.orchestra.run.smithi156:> sudo lsb_release -is 2020-03-28T06:15:03.199 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS 2020-03-28T06:15:03.202 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-28T06:15:03.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=cd131a1eb0d13cbc9e432c4911cf5025d18aef1e 2020-03-28T06:15:03.214 INFO:teuthology.orchestra.run.smithi168.stdout:CentOS 2020-03-28T06:15:03.216 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-28T06:15:03.216 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'notcmalloc', 'extras': None} 2020-03-28T06:15:03.217 INFO:teuthology.orchestra.run.smithi168:> sudo lsb_release -is 2020-03-28T06:15:03.349 INFO:teuthology.orchestra.run.smithi168.stdout:CentOS 2020-03-28T06:15:03.351 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-28T06:15:03.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=cd131a1eb0d13cbc9e432c4911cf5025d18aef1e 2020-03-28T06:15:03.439 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/ 2020-03-28T06:15:03.439 INFO:teuthology.task.install.rpm:Package version is 14.2.8-334.gcd131a1eb0 2020-03-28T06:15:03.589 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/ 2020-03-28T06:15:03.589 INFO:teuthology.task.install.rpm:Package version is 14.2.8-334.gcd131a1eb0 2020-03-28T06:15:03.823 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-28T06:15:03.823 INFO:teuthology.orchestra.run.smithi156:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-28T06:15:03.878 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/nautilus-kefu2-testing-2020-03-27-2234/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/centos/7/flavors/notcmalloc/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-28T06:15:03.879 INFO:teuthology.orchestra.run.smithi168:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-28T06:15:03.941 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2020-03-28T06:15:03.942 INFO:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-28T06:15:03.998 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2020-03-28T06:15:03.998 INFO:teuthology.orchestra.run.smithi168:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cd131a1eb0d13cbc9e432c4911cf5025d18aef1e/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-28T06:15:04.049 INFO:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-28T06:15:04.106 INFO:teuthology.orchestra.run.smithi168:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-28T06:15:04.157 INFO:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-28T06:15:04.231 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2020-03-28T06:15:04.233 INFO:teuthology.orchestra.run.smithi156:> sudo yum clean all 2020-03-28T06:15:04.239 INFO:teuthology.orchestra.run.smithi168:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-28T06:15:04.324 INFO:teuthology.orchestra.run.smithi168.stdout:check_obsoletes = 1 2020-03-28T06:15:04.326 INFO:teuthology.orchestra.run.smithi168:> sudo yum clean all 2020-03-28T06:15:04.391 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:04.429 INFO:teuthology.orchestra.run.smithi156.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2020-03-28T06:15:04.429 INFO:teuthology.orchestra.run.smithi156.stdout: : lab-extras updates 2020-03-28T06:15:04.451 INFO:teuthology.orchestra.run.smithi156.stdout:Cleaning up list of fastest mirrors 2020-03-28T06:15:04.464 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw 2020-03-28T06:15:04.509 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:04.547 INFO:teuthology.orchestra.run.smithi168.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2020-03-28T06:15:04.547 INFO:teuthology.orchestra.run.smithi168.stdout: : lab-extras updates 2020-03-28T06:15:04.568 INFO:teuthology.orchestra.run.smithi168.stdout:Cleaning up list of fastest mirrors 2020-03-28T06:15:04.582 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-radosgw 2020-03-28T06:15:04.659 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:04.698 INFO:teuthology.orchestra.run.smithi156.stdout:Determining fastest mirrors 2020-03-28T06:15:04.758 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:04.795 INFO:teuthology.orchestra.run.smithi168.stdout:Determining fastest mirrors 2020-03-28T06:15:04.981 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:15:04.982 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:15:04.983 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:15:04.983 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:15:06.653 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:15:06.654 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:15:06.655 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:15:06.655 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:15:13.077 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:15:13.437 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:15:13.437 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:13.438 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-radosgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.452 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: librgw2 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.564 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: librados2 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.565 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.566 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-base = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.568 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.568 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.570 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.571 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libceph-common.so.0()(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.572 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:13.572 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-base.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.582 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: librbd1 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.583 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-common = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.584 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.585 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-requests for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.587 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.588 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.589 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-selinux.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.592 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2020-03-28T06:15:13.593 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2020-03-28T06:15:13.593 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2020-03-28T06:15:13.593 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librados2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be an update 2020-03-28T06:15:13.595 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librgw2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.597 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:13.597 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-common.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.601 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-rados = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.602 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-cephfs = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.603 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.604 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libradosstriper1 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.605 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-rgw = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.606 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-rbd = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.607 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.607 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.609 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.610 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.611 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:13.611 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2020-03-28T06:15:13.611 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package libcephfs2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.612 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2020-03-28T06:15:13.612 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librbd1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be an update 2020-03-28T06:15:13.613 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2020-03-28T06:15:13.614 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2020-03-28T06:15:13.614 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2020-03-28T06:15:13.615 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-requests.noarch 0:2.6.0-9.el7_7 will be installed 2020-03-28T06:15:13.615 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-9.el7_7.noarch 2020-03-28T06:15:13.616 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:13.616 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2020-03-28T06:15:13.617 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package libradosstriper1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.618 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2020-03-28T06:15:13.620 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.620 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-cephfs.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.622 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2020-03-28T06:15:13.622 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-rados.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.622 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-rbd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.622 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-rgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:13.622 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2020-03-28T06:15:13.623 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2020-03-28T06:15:14.499 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:15:14.571 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:15:14.586 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:14.586 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:15:14.589 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:14.589 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:14.590 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository 2020-03-28T06:15:14.590 INFO:teuthology.orchestra.run.smithi156.stdout: Size 2020-03-28T06:15:14.590 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:14.590 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:15:14.590 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 5.8 M 2020-03-28T06:15:14.590 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:15:14.590 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 5.4 M 2020-03-28T06:15:14.591 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 19 M 2020-03-28T06:15:14.591 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 20 k 2020-03-28T06:15:14.591 INFO:teuthology.orchestra.run.smithi156.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2020-03-28T06:15:14.591 INFO:teuthology.orchestra.run.smithi156.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2020-03-28T06:15:14.591 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 500 k 2020-03-28T06:15:14.591 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2020-03-28T06:15:14.591 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2020-03-28T06:15:14.592 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 363 k 2020-03-28T06:15:14.592 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2020-03-28T06:15:14.592 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 5.1 M 2020-03-28T06:15:14.592 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2020-03-28T06:15:14.592 INFO:teuthology.orchestra.run.smithi156.stdout: python-ceph-argparse x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 35 k 2020-03-28T06:15:14.592 INFO:teuthology.orchestra.run.smithi156.stdout: python-cephfs x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 111 k 2020-03-28T06:15:14.592 INFO:teuthology.orchestra.run.smithi156.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2020-03-28T06:15:14.593 INFO:teuthology.orchestra.run.smithi156.stdout: python-rados x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 198 k 2020-03-28T06:15:14.593 INFO:teuthology.orchestra.run.smithi156.stdout: python-rbd x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 180 k 2020-03-28T06:15:14.593 INFO:teuthology.orchestra.run.smithi156.stdout: python-requests noarch 2.6.0-9.el7_7 updates 94 k 2020-03-28T06:15:14.593 INFO:teuthology.orchestra.run.smithi156.stdout: python-rgw x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 77 k 2020-03-28T06:15:14.593 INFO:teuthology.orchestra.run.smithi156.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout:Updating for dependencies: 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 3.5 M 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 1.7 M 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:14.594 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+21 Dependent packages) 2020-03-28T06:15:14.595 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade ( 2 Dependent packages) 2020-03-28T06:15:14.595 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:14.595 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 43 M 2020-03-28T06:15:14.595 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:15:14.595 INFO:teuthology.orchestra.run.smithi156.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-28T06:15:14.930 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:15:14.931 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:14.931 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-radosgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:14.946 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: librgw2 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.059 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: librados2 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.060 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.061 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-base = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.063 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.063 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.065 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.066 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libceph-common.so.0()(64bit) for package: 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.067 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:15.067 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-base.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.076 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: librbd1 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.077 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-common = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.078 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libcephfs2 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.079 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-requests for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.082 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.083 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.083 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-selinux.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.086 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2020-03-28T06:15:15.087 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2020-03-28T06:15:15.087 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2020-03-28T06:15:15.088 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librados2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be an update 2020-03-28T06:15:15.090 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librgw2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.091 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:15.092 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-common.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.095 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-rados = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.096 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-cephfs = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.097 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.098 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libradosstriper1 = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.099 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-rgw = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.100 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-rbd = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.101 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.102 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.103 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.104 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.105 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:15.105 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2020-03-28T06:15:15.105 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package libcephfs2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.106 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2020-03-28T06:15:15.106 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librbd1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be an update 2020-03-28T06:15:15.107 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2020-03-28T06:15:15.108 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2020-03-28T06:15:15.109 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2020-03-28T06:15:15.109 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-requests.noarch 0:2.6.0-9.el7_7 will be installed 2020-03-28T06:15:15.109 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-9.el7_7.noarch 2020-03-28T06:15:15.110 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:15.110 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2020-03-28T06:15:15.112 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package libradosstriper1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.113 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2020-03-28T06:15:15.113 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.114 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-cephfs.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.115 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2020-03-28T06:15:15.115 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-rados.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.115 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-rbd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.116 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-rgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:15.116 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2020-03-28T06:15:15.117 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2020-03-28T06:15:15.990 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:15:16.076 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:16.077 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:15:16.080 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:16.080 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:16.080 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository 2020-03-28T06:15:16.080 INFO:teuthology.orchestra.run.smithi168.stdout: Size 2020-03-28T06:15:16.080 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-radosgw x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 5.8 M 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-base x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 5.4 M 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-common x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 19 M 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-selinux x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 20 k 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2020-03-28T06:15:16.081 INFO:teuthology.orchestra.run.smithi168.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2020-03-28T06:15:16.082 INFO:teuthology.orchestra.run.smithi168.stdout: libcephfs2 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 500 k 2020-03-28T06:15:16.082 INFO:teuthology.orchestra.run.smithi168.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2020-03-28T06:15:16.082 INFO:teuthology.orchestra.run.smithi168.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2020-03-28T06:15:16.082 INFO:teuthology.orchestra.run.smithi168.stdout: libradosstriper1 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 363 k 2020-03-28T06:15:16.082 INFO:teuthology.orchestra.run.smithi168.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2020-03-28T06:15:16.082 INFO:teuthology.orchestra.run.smithi168.stdout: librgw2 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 5.1 M 2020-03-28T06:15:16.082 INFO:teuthology.orchestra.run.smithi168.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2020-03-28T06:15:16.083 INFO:teuthology.orchestra.run.smithi168.stdout: python-ceph-argparse x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 35 k 2020-03-28T06:15:16.083 INFO:teuthology.orchestra.run.smithi168.stdout: python-cephfs x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 111 k 2020-03-28T06:15:16.083 INFO:teuthology.orchestra.run.smithi168.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2020-03-28T06:15:16.083 INFO:teuthology.orchestra.run.smithi168.stdout: python-rados x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 198 k 2020-03-28T06:15:16.083 INFO:teuthology.orchestra.run.smithi168.stdout: python-rbd x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 180 k 2020-03-28T06:15:16.083 INFO:teuthology.orchestra.run.smithi168.stdout: python-requests noarch 2.6.0-9.el7_7 updates 94 k 2020-03-28T06:15:16.084 INFO:teuthology.orchestra.run.smithi168.stdout: python-rgw x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 77 k 2020-03-28T06:15:16.084 INFO:teuthology.orchestra.run.smithi168.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2020-03-28T06:15:16.084 INFO:teuthology.orchestra.run.smithi168.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2020-03-28T06:15:16.084 INFO:teuthology.orchestra.run.smithi168.stdout:Updating for dependencies: 2020-03-28T06:15:16.084 INFO:teuthology.orchestra.run.smithi168.stdout: librados2 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 3.5 M 2020-03-28T06:15:16.084 INFO:teuthology.orchestra.run.smithi168.stdout: librbd1 x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 1.7 M 2020-03-28T06:15:16.084 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:16.085 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:15:16.085 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:16.085 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+21 Dependent packages) 2020-03-28T06:15:16.085 INFO:teuthology.orchestra.run.smithi168.stdout:Upgrade ( 2 Dependent packages) 2020-03-28T06:15:16.085 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:16.085 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 43 M 2020-03-28T06:15:16.085 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:15:16.086 INFO:teuthology.orchestra.run.smithi168.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-03-28T06:15:16.816 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:15:16.816 INFO:teuthology.orchestra.run.smithi156.stdout:Total 19 MB/s | 43 MB 00:02 2020-03-28T06:15:16.830 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:15:16.889 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:15:17.124 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:15:17.125 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:15:17.793 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/26 2020-03-28T06:15:18.071 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/26 2020-03-28T06:15:18.336 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:python-ceph-argparse-14.2.8-334.gcd131a1eb0.el7.x86_64 3/26 2020-03-28T06:15:18.534 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:15:18.534 INFO:teuthology.orchestra.run.smithi168.stdout:Total 18 MB/s | 43 MB 00:02 2020-03-28T06:15:18.553 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:15:18.628 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:15:18.706 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 4/26 2020-03-28T06:15:18.883 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:15:18.888 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:15:19.038 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 5/26 2020-03-28T06:15:19.630 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/26 2020-03-28T06:15:19.777 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 6/26 2020-03-28T06:15:19.874 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/26 2020-03-28T06:15:20.127 INFO:teuthology.orchestra.run.smithi156.stdout: Updating : 2:librados2-14.2.8-334.gcd131a1eb0.el7.x86_64 7/26 2020-03-28T06:15:20.148 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:python-ceph-argparse-14.2.8-334.gcd131a1eb0.el7.x86_64 3/26 2020-03-28T06:15:20.443 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 4/26 2020-03-28T06:15:20.836 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 5/26 2020-03-28T06:15:20.955 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:python-rados-14.2.8-334.gcd131a1eb0.el7.x86_64 8/26 2020-03-28T06:15:21.492 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:librgw2-14.2.8-334.gcd131a1eb0.el7.x86_64 9/26 2020-03-28T06:15:21.636 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 6/26 2020-03-28T06:15:21.861 INFO:teuthology.orchestra.run.smithi156.stdout: Updating : 2:librbd1-14.2.8-334.gcd131a1eb0.el7.x86_64 10/26 2020-03-28T06:15:22.094 INFO:teuthology.orchestra.run.smithi168.stdout: Updating : 2:librados2-14.2.8-334.gcd131a1eb0.el7.x86_64 7/26 2020-03-28T06:15:22.186 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:libcephfs2-14.2.8-334.gcd131a1eb0.el7.x86_64 11/26 2020-03-28T06:15:22.441 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:python-cephfs-14.2.8-334.gcd131a1eb0.el7.x86_64 12/26 2020-03-28T06:15:22.709 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:python-rbd-14.2.8-334.gcd131a1eb0.el7.x86_64 13/26 2020-03-28T06:15:22.881 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:python-rados-14.2.8-334.gcd131a1eb0.el7.x86_64 8/26 2020-03-28T06:15:23.065 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:python-rgw-14.2.8-334.gcd131a1eb0.el7.x86_64 14/26 2020-03-28T06:15:23.375 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:libradosstriper1-14.2.8-334.gcd131a1eb0.el7.x86_64 15/26 2020-03-28T06:15:23.411 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:librgw2-14.2.8-334.gcd131a1eb0.el7.x86_64 9/26 2020-03-28T06:15:23.719 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 16/26 2020-03-28T06:15:23.777 INFO:teuthology.orchestra.run.smithi168.stdout: Updating : 2:librbd1-14.2.8-334.gcd131a1eb0.el7.x86_64 10/26 2020-03-28T06:15:24.050 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 17/26 2020-03-28T06:15:24.090 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:libcephfs2-14.2.8-334.gcd131a1eb0.el7.x86_64 11/26 2020-03-28T06:15:24.370 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:python-cephfs-14.2.8-334.gcd131a1eb0.el7.x86_64 12/26 2020-03-28T06:15:24.406 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 18/26 2020-03-28T06:15:24.656 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:python-rbd-14.2.8-334.gcd131a1eb0.el7.x86_64 13/26 2020-03-28T06:15:24.768 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 19/26 2020-03-28T06:15:24.984 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:python-rgw-14.2.8-334.gcd131a1eb0.el7.x86_64 14/26 2020-03-28T06:15:25.304 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:libradosstriper1-14.2.8-334.gcd131a1eb0.el7.x86_64 15/26 2020-03-28T06:15:25.643 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 16/26 2020-03-28T06:15:25.946 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 17/26 2020-03-28T06:15:26.295 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 18/26 2020-03-28T06:15:26.669 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 19/26 2020-03-28T06:15:27.083 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-requests-2.6.0-9.el7_7.noarch 20/26 2020-03-28T06:15:28.242 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 21/26 2020-03-28T06:15:28.746 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 22/26 2020-03-28T06:15:29.050 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-requests-2.6.0-9.el7_7.noarch 20/26 2020-03-28T06:15:30.141 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 21/26 2020-03-28T06:15:30.669 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 22/26 2020-03-28T06:15:38.284 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-selinux-14.2.8-334.gcd131a1eb0.el7.x86_64 23/26 2020-03-28T06:15:38.983 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 24/26 2020-03-28T06:15:39.252 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 25/26 2020-03-28T06:15:39.557 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 26/26 2020-03-28T06:15:39.664 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 1/26 2020-03-28T06:15:39.807 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-selinux-14.2.8-334.gcd131a1eb0.el7.x86_64 2/26 2020-03-28T06:15:39.902 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-selinux-14.2.8-334.gcd131a1eb0.el7.x86_64 23/26 2020-03-28T06:15:39.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 3/26 2020-03-28T06:15:40.024 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:librgw2-14.2.8-334.gcd131a1eb0.el7.x86_64 4/26 2020-03-28T06:15:40.124 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 5/26 2020-03-28T06:15:40.233 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:python-rgw-14.2.8-334.gcd131a1eb0.el7.x86_64 6/26 2020-03-28T06:15:40.256 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 24/26 2020-03-28T06:15:40.350 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/26 2020-03-28T06:15:40.442 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 8/26 2020-03-28T06:15:40.498 INFO:teuthology.orchestra.run.smithi168.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 25/26 2020-03-28T06:15:40.534 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:libradosstriper1-14.2.8-334.gcd131a1eb0.el7.x86_64 9/26 2020-03-28T06:15:40.660 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 10/26 2020-03-28T06:15:40.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:python-rbd-14.2.8-334.gcd131a1eb0.el7.x86_64 11/26 2020-03-28T06:15:40.783 INFO:teuthology.orchestra.run.smithi168.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 26/26 2020-03-28T06:15:40.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:python-ceph-argparse-14.2.8-334.gcd131a1eb0.el7.x86_64 12/26 2020-03-28T06:15:40.970 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 1/26 2020-03-28T06:15:40.978 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-requests-2.6.0-9.el7_7.noarch 13/26 2020-03-28T06:15:41.079 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 14/26 2020-03-28T06:15:41.086 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-selinux-14.2.8-334.gcd131a1eb0.el7.x86_64 2/26 2020-03-28T06:15:41.179 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:librados2-14.2.8-334.gcd131a1eb0.el7.x86_64 15/26 2020-03-28T06:15:41.196 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 3/26 2020-03-28T06:15:41.280 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 16/26 2020-03-28T06:15:41.304 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:librgw2-14.2.8-334.gcd131a1eb0.el7.x86_64 4/26 2020-03-28T06:15:41.364 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:librbd1-14.2.8-334.gcd131a1eb0.el7.x86_64 17/26 2020-03-28T06:15:41.438 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-base-14.2.8-334.gcd131a1eb0.el7.x86_64 5/26 2020-03-28T06:15:41.490 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 18/26 2020-03-28T06:15:41.607 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:libcephfs2-14.2.8-334.gcd131a1eb0.el7.x86_64 19/26 2020-03-28T06:15:41.707 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 20/26 2020-03-28T06:15:41.755 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:python-rgw-14.2.8-334.gcd131a1eb0.el7.x86_64 6/26 2020-03-28T06:15:41.808 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 21/26 2020-03-28T06:15:41.902 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/26 2020-03-28T06:15:41.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:python-rados-14.2.8-334.gcd131a1eb0.el7.x86_64 22/26 2020-03-28T06:15:41.998 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 8/26 2020-03-28T06:15:42.028 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 23/26 2020-03-28T06:15:42.082 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:libradosstriper1-14.2.8-334.gcd131a1eb0.el7.x86_64 9/26 2020-03-28T06:15:42.134 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:python-cephfs-14.2.8-334.gcd131a1eb0.el7.x86_64 24/26 2020-03-28T06:15:42.136 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 25/26 2020-03-28T06:15:42.191 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-radosgw-14.2.8-334.gcd131a1eb0.el7.x86_64 10/26 2020-03-28T06:15:42.292 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:python-rbd-14.2.8-334.gcd131a1eb0.el7.x86_64 11/26 2020-03-28T06:15:42.425 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:python-ceph-argparse-14.2.8-334.gcd131a1eb0.el7.x86_64 12/26 2020-03-28T06:15:42.533 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 26/26 2020-03-28T06:15:42.533 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:42.533 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:15:42.533 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.533 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout: leveldb.x86_64 0:1.12.0-11.el7 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.534 INFO:teuthology.orchestra.run.smithi156.stdout: liboath.x86_64 0:2.6.2-1.el7 2020-03-28T06:15:42.535 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2020-03-28T06:15:42.535 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.535 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2020-03-28T06:15:42.535 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.535 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2020-03-28T06:15:42.535 INFO:teuthology.orchestra.run.smithi156.stdout: python-ceph-argparse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.535 INFO:teuthology.orchestra.run.smithi156.stdout: python-cephfs.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.536 INFO:teuthology.orchestra.run.smithi156.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2020-03-28T06:15:42.536 INFO:teuthology.orchestra.run.smithi156.stdout: python-rados.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.536 INFO:teuthology.orchestra.run.smithi156.stdout: python-rbd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.536 INFO:teuthology.orchestra.run.smithi156.stdout: python-requests.noarch 0:2.6.0-9.el7_7 2020-03-28T06:15:42.536 INFO:teuthology.orchestra.run.smithi156.stdout: python-rgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.536 INFO:teuthology.orchestra.run.smithi156.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2020-03-28T06:15:42.536 INFO:teuthology.orchestra.run.smithi156.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2020-03-28T06:15:42.537 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:42.537 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Updated: 2020-03-28T06:15:42.537 INFO:teuthology.orchestra.run.smithi156.stdout: librados2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.537 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:42.537 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:42.537 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:15:42.551 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-requests-2.6.0-9.el7_7.noarch 13/26 2020-03-28T06:15:42.651 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 14/26 2020-03-28T06:15:42.656 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-test 2020-03-28T06:15:42.768 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:librados2-14.2.8-334.gcd131a1eb0.el7.x86_64 15/26 2020-03-28T06:15:42.885 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:42.908 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 16/26 2020-03-28T06:15:42.926 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:15:42.927 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:15:42.927 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:15:42.928 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:15:42.928 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:15:43.031 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:librbd1-14.2.8-334.gcd131a1eb0.el7.x86_64 17/26 2020-03-28T06:15:43.139 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 18/26 2020-03-28T06:15:43.246 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:libcephfs2-14.2.8-334.gcd131a1eb0.el7.x86_64 19/26 2020-03-28T06:15:43.347 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 20/26 2020-03-28T06:15:43.481 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-common-14.2.8-334.gcd131a1eb0.el7.x86_64 21/26 2020-03-28T06:15:43.576 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:15:43.598 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:python-rados-14.2.8-334.gcd131a1eb0.el7.x86_64 22/26 2020-03-28T06:15:43.698 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 23/26 2020-03-28T06:15:43.807 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:python-cephfs-14.2.8-334.gcd131a1eb0.el7.x86_64 24/26 2020-03-28T06:15:43.809 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 25/26 2020-03-28T06:15:43.912 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:15:43.913 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:43.913 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-test.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:43.933 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:43.963 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:43.964 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:43.965 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:43.965 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package jq.x86_64 0:1.6-1.el7 will be installed 2020-03-28T06:15:43.966 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.6-1.el7.x86_64 2020-03-28T06:15:43.966 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2020-03-28T06:15:43.968 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2020-03-28T06:15:43.971 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:43.971 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2020-03-28T06:15:44.173 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 26/26 2020-03-28T06:15:44.174 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:44.174 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:15:44.174 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-radosgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.174 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:44.175 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:15:44.175 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-base.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.175 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-common.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.175 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-selinux.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.176 INFO:teuthology.orchestra.run.smithi168.stdout: leveldb.x86_64 0:1.12.0-11.el7 2020-03-28T06:15:44.176 INFO:teuthology.orchestra.run.smithi168.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2020-03-28T06:15:44.176 INFO:teuthology.orchestra.run.smithi168.stdout: libcephfs2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.176 INFO:teuthology.orchestra.run.smithi168.stdout: liboath.x86_64 0:2.6.2-1.el7 2020-03-28T06:15:44.177 INFO:teuthology.orchestra.run.smithi168.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2020-03-28T06:15:44.177 INFO:teuthology.orchestra.run.smithi168.stdout: libradosstriper1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.177 INFO:teuthology.orchestra.run.smithi168.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2020-03-28T06:15:44.177 INFO:teuthology.orchestra.run.smithi168.stdout: librgw2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.178 INFO:teuthology.orchestra.run.smithi168.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2020-03-28T06:15:44.178 INFO:teuthology.orchestra.run.smithi168.stdout: python-ceph-argparse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.178 INFO:teuthology.orchestra.run.smithi168.stdout: python-cephfs.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.178 INFO:teuthology.orchestra.run.smithi168.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2020-03-28T06:15:44.178 INFO:teuthology.orchestra.run.smithi168.stdout: python-rados.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.179 INFO:teuthology.orchestra.run.smithi168.stdout: python-rbd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.179 INFO:teuthology.orchestra.run.smithi168.stdout: python-requests.noarch 0:2.6.0-9.el7_7 2020-03-28T06:15:44.179 INFO:teuthology.orchestra.run.smithi168.stdout: python-rgw.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.179 INFO:teuthology.orchestra.run.smithi168.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2020-03-28T06:15:44.180 INFO:teuthology.orchestra.run.smithi168.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2020-03-28T06:15:44.180 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:44.180 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Updated: 2020-03-28T06:15:44.180 INFO:teuthology.orchestra.run.smithi168.stdout: librados2.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.180 INFO:teuthology.orchestra.run.smithi168.stdout: librbd1.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:44.181 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:44.181 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:15:44.296 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-test 2020-03-28T06:15:44.333 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:15:44.559 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:44.560 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:15:44.561 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:44.561 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:44.561 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository 2020-03-28T06:15:44.562 INFO:teuthology.orchestra.run.smithi156.stdout: Size 2020-03-28T06:15:44.562 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:44.562 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:15:44.562 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 34 M 2020-03-28T06:15:44.562 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:15:44.562 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-1.el7 epel 167 k 2020-03-28T06:15:44.563 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2020-03-28T06:15:44.563 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2020-03-28T06:15:44.563 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2020-03-28T06:15:44.563 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:44.563 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:15:44.563 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:44.564 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+4 Dependent packages) 2020-03-28T06:15:44.564 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:44.564 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 35 M 2020-03-28T06:15:44.564 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 120 M 2020-03-28T06:15:44.564 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:15:44.577 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:44.619 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:15:44.620 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:15:44.621 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:15:44.621 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:15:44.621 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:15:45.281 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:15:45.620 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:15:45.620 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:45.620 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-test.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:45.640 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:45.670 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:45.671 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:45.673 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:45.673 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package jq.x86_64 0:1.6-1.el7 will be installed 2020-03-28T06:15:45.673 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.6-1.el7.x86_64 2020-03-28T06:15:45.674 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2020-03-28T06:15:45.676 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2020-03-28T06:15:45.679 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:45.679 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2020-03-28T06:15:46.042 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:15:46.270 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:46.270 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:15:46.271 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:46.271 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:46.271 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository 2020-03-28T06:15:46.271 INFO:teuthology.orchestra.run.smithi168.stdout: Size 2020-03-28T06:15:46.271 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:46.271 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:15:46.272 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-test x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 34 M 2020-03-28T06:15:46.272 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:15:46.272 INFO:teuthology.orchestra.run.smithi168.stdout: jq x86_64 1.6-1.el7 epel 167 k 2020-03-28T06:15:46.272 INFO:teuthology.orchestra.run.smithi168.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2020-03-28T06:15:46.272 INFO:teuthology.orchestra.run.smithi168.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2020-03-28T06:15:46.272 INFO:teuthology.orchestra.run.smithi168.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2020-03-28T06:15:46.272 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:46.273 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:15:46.273 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:46.273 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+4 Dependent packages) 2020-03-28T06:15:46.273 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:46.274 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 35 M 2020-03-28T06:15:46.274 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 120 M 2020-03-28T06:15:46.274 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:15:46.444 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:15:46.444 INFO:teuthology.orchestra.run.smithi156.stdout:Total 19 MB/s | 35 MB 00:01 2020-03-28T06:15:46.447 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:15:46.472 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:15:46.526 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:15:46.527 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:15:47.201 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2020-03-28T06:15:47.473 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2020-03-28T06:15:47.774 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2020-03-28T06:15:48.175 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:15:48.175 INFO:teuthology.orchestra.run.smithi168.stdout:Total 18 MB/s | 35 MB 00:01 2020-03-28T06:15:48.177 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:15:48.197 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:15:48.245 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:15:48.245 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:15:48.961 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2020-03-28T06:15:49.231 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2020-03-28T06:15:49.513 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2020-03-28T06:15:51.521 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-1.el7.x86_64 4/5 2020-03-28T06:15:51.859 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 5/5 2020-03-28T06:15:51.967 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 1/5 2020-03-28T06:15:52.068 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 2/5 2020-03-28T06:15:52.177 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-1.el7.x86_64 3/5 2020-03-28T06:15:52.310 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 4/5 2020-03-28T06:15:52.745 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2020-03-28T06:15:52.746 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:52.746 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:15:52.746 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:52.746 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:52.746 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:15:52.746 INFO:teuthology.orchestra.run.smithi156.stdout: jq.x86_64 0:1.6-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2020-03-28T06:15:52.747 INFO:teuthology.orchestra.run.smithi156.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2020-03-28T06:15:52.747 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:52.747 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:15:52.868 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph 2020-03-28T06:15:53.147 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:53.192 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : jq-1.6-1.el7.x86_64 4/5 2020-03-28T06:15:53.193 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:15:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:15:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:15:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:15:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:15:53.524 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 5/5 2020-03-28T06:15:53.649 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 1/5 2020-03-28T06:15:53.749 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 2/5 2020-03-28T06:15:53.882 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:15:53.892 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : jq-1.6-1.el7.x86_64 3/5 2020-03-28T06:15:54.000 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-test-14.2.8-334.gcd131a1eb0.el7.x86_64 4/5 2020-03-28T06:15:54.228 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:15:54.229 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:54.229 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:54.231 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.261 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-mds = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.262 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-mon = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.263 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-osd = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.264 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:54.265 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mds.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:54.276 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mgr.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:54.279 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.280 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.281 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.281 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.282 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:54.283 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mon.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:54.286 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-osd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:54.291 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:54.291 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2020-03-28T06:15:54.292 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2020-03-28T06:15:54.293 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2020-03-28T06:15:54.294 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:54.294 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:54.295 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:54.296 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:54.297 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:54.298 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2020-03-28T06:15:54.298 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2020-03-28T06:15:54.298 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2020-03-28T06:15:54.299 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:54.300 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2020-03-28T06:15:54.300 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2020-03-28T06:15:54.300 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2020-03-28T06:15:54.302 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2020-03-28T06:15:54.302 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2020-03-28T06:15:54.302 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2020-03-28T06:15:54.302 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2020-03-28T06:15:54.302 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2020-03-28T06:15:54.302 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:54.302 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2020-03-28T06:15:54.303 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2020-03-28T06:15:54.303 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2020-03-28T06:15:54.303 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:54.304 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2020-03-28T06:15:54.304 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2020-03-28T06:15:54.304 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:15:54.304 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2020-03-28T06:15:54.494 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2020-03-28T06:15:54.494 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:54.495 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:15:54.495 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-test.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:15:54.495 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:54.495 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:15:54.495 INFO:teuthology.orchestra.run.smithi168.stdout: jq.x86_64 0:1.6-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2020-03-28T06:15:54.495 INFO:teuthology.orchestra.run.smithi168.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2020-03-28T06:15:54.496 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:54.496 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:15:54.623 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph 2020-03-28T06:15:54.663 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:15:54.844 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:15:54.882 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:15:54.883 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:15:54.884 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:15:54.884 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:15:54.884 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:15:54.893 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:54.893 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:15:54.895 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:54.896 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:54.896 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository 2020-03-28T06:15:54.896 INFO:teuthology.orchestra.run.smithi156.stdout: Size 2020-03-28T06:15:54.896 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:54.897 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:15:54.897 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 2.0 k 2020-03-28T06:15:54.897 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:15:54.897 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 1.9 M 2020-03-28T06:15:54.898 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 1.7 M 2020-03-28T06:15:54.898 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 4.3 M 2020-03-28T06:15:54.898 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 16 M 2020-03-28T06:15:54.898 INFO:teuthology.orchestra.run.smithi156.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2020-03-28T06:15:54.898 INFO:teuthology.orchestra.run.smithi156.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2020-03-28T06:15:54.898 INFO:teuthology.orchestra.run.smithi156.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2020-03-28T06:15:54.899 INFO:teuthology.orchestra.run.smithi156.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2020-03-28T06:15:54.899 INFO:teuthology.orchestra.run.smithi156.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2020-03-28T06:15:54.899 INFO:teuthology.orchestra.run.smithi156.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2020-03-28T06:15:54.899 INFO:teuthology.orchestra.run.smithi156.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2020-03-28T06:15:54.899 INFO:teuthology.orchestra.run.smithi156.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2020-03-28T06:15:54.899 INFO:teuthology.orchestra.run.smithi156.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2020-03-28T06:15:54.900 INFO:teuthology.orchestra.run.smithi156.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2020-03-28T06:15:54.900 INFO:teuthology.orchestra.run.smithi156.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2020-03-28T06:15:54.900 INFO:teuthology.orchestra.run.smithi156.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2020-03-28T06:15:54.900 INFO:teuthology.orchestra.run.smithi156.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2020-03-28T06:15:54.900 INFO:teuthology.orchestra.run.smithi156.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2020-03-28T06:15:54.900 INFO:teuthology.orchestra.run.smithi156.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2020-03-28T06:15:54.900 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:54.901 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:15:54.901 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:15:54.901 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+19 Dependent packages) 2020-03-28T06:15:54.901 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:15:54.902 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 27 M 2020-03-28T06:15:54.902 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 86 M 2020-03-28T06:15:54.902 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:15:55.537 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:15:55.872 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:15:55.872 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:55.872 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:55.874 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-mgr = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.904 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-mds = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.906 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-mon = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.907 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-osd = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.908 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:55.909 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mds.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:55.920 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mgr.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:55.922 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.923 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.924 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.925 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.926 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:15:55.927 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mon.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:55.930 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-osd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:15:55.935 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:55.935 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2020-03-28T06:15:55.936 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2020-03-28T06:15:55.937 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2020-03-28T06:15:55.937 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:55.938 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:55.939 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:55.940 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:55.941 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2020-03-28T06:15:55.942 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2020-03-28T06:15:55.942 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2020-03-28T06:15:55.943 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2020-03-28T06:15:55.943 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:55.944 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2020-03-28T06:15:55.944 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2020-03-28T06:15:55.944 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2020-03-28T06:15:55.946 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2020-03-28T06:15:55.946 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2020-03-28T06:15:55.946 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2020-03-28T06:15:55.946 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2020-03-28T06:15:55.947 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2020-03-28T06:15:55.947 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:55.947 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2020-03-28T06:15:55.947 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2020-03-28T06:15:55.948 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2020-03-28T06:15:55.948 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:55.948 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2020-03-28T06:15:55.949 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2020-03-28T06:15:55.949 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:15:55.949 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2020-03-28T06:15:56.312 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:15:56.435 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:15:56.436 INFO:teuthology.orchestra.run.smithi156.stdout:Total 18 MB/s | 27 MB 00:01 2020-03-28T06:15:56.449 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:15:56.481 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:15:56.547 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:56.547 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:15:56.550 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:56.550 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:56.550 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository 2020-03-28T06:15:56.550 INFO:teuthology.orchestra.run.smithi168.stdout: Size 2020-03-28T06:15:56.551 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:56.551 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:15:56.551 INFO:teuthology.orchestra.run.smithi168.stdout: ceph x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 2.0 k 2020-03-28T06:15:56.551 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:15:56.551 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mds x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 1.9 M 2020-03-28T06:15:56.551 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 1.7 M 2020-03-28T06:15:56.551 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mon x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 4.3 M 2020-03-28T06:15:56.552 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-osd x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 16 M 2020-03-28T06:15:56.552 INFO:teuthology.orchestra.run.smithi168.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2020-03-28T06:15:56.552 INFO:teuthology.orchestra.run.smithi168.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2020-03-28T06:15:56.552 INFO:teuthology.orchestra.run.smithi168.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2020-03-28T06:15:56.552 INFO:teuthology.orchestra.run.smithi168.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2020-03-28T06:15:56.552 INFO:teuthology.orchestra.run.smithi168.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2020-03-28T06:15:56.552 INFO:teuthology.orchestra.run.smithi168.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2020-03-28T06:15:56.553 INFO:teuthology.orchestra.run.smithi168.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2020-03-28T06:15:56.553 INFO:teuthology.orchestra.run.smithi168.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2020-03-28T06:15:56.553 INFO:teuthology.orchestra.run.smithi168.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2020-03-28T06:15:56.553 INFO:teuthology.orchestra.run.smithi168.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2020-03-28T06:15:56.554 INFO:teuthology.orchestra.run.smithi168.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2020-03-28T06:15:56.554 INFO:teuthology.orchestra.run.smithi168.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2020-03-28T06:15:56.554 INFO:teuthology.orchestra.run.smithi168.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2020-03-28T06:15:56.554 INFO:teuthology.orchestra.run.smithi168.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2020-03-28T06:15:56.554 INFO:teuthology.orchestra.run.smithi168.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2020-03-28T06:15:56.554 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:56.554 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:15:56.555 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:15:56.555 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+19 Dependent packages) 2020-03-28T06:15:56.555 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:15:56.555 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 27 M 2020-03-28T06:15:56.556 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 86 M 2020-03-28T06:15:56.556 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:15:56.752 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:15:56.753 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:15:57.524 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 1/20 2020-03-28T06:15:57.916 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-webob-1.2.3-7.el7.noarch 2/20 2020-03-28T06:15:58.073 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:15:58.073 INFO:teuthology.orchestra.run.smithi168.stdout:Total 18 MB/s | 27 MB 00:01 2020-03-28T06:15:58.085 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:15:58.112 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:15:58.446 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:15:58.446 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:15:59.214 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 1/20 2020-03-28T06:15:59.236 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/20 2020-03-28T06:15:59.563 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-webob-1.2.3-7.el7.noarch 2/20 2020-03-28T06:15:59.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mds-14.2.8-334.gcd131a1eb0.el7.x86_64 4/20 2020-03-28T06:16:00.099 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/20 2020-03-28T06:16:00.489 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mds-14.2.8-334.gcd131a1eb0.el7.x86_64 4/20 2020-03-28T06:16:01.439 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 5/20 2020-03-28T06:16:01.818 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-osd-14.2.8-334.gcd131a1eb0.el7.x86_64 6/20 2020-03-28T06:16:02.302 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 5/20 2020-03-28T06:16:02.326 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 7/20 2020-03-28T06:16:02.687 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-osd-14.2.8-334.gcd131a1eb0.el7.x86_64 6/20 2020-03-28T06:16:02.721 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 8/20 2020-03-28T06:16:03.155 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 7/20 2020-03-28T06:16:03.225 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 9/20 2020-03-28T06:16:03.600 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 8/20 2020-03-28T06:16:03.631 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 10/20 2020-03-28T06:16:03.861 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-six-1.9.0-0.el7.noarch 11/20 2020-03-28T06:16:04.078 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 9/20 2020-03-28T06:16:04.346 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 12/20 2020-03-28T06:16:04.365 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 10/20 2020-03-28T06:16:04.627 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-six-1.9.0-0.el7.noarch 11/20 2020-03-28T06:16:05.087 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 12/20 2020-03-28T06:16:05.177 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 13/20 2020-03-28T06:16:05.544 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mon-14.2.8-334.gcd131a1eb0.el7.x86_64 14/20 2020-03-28T06:16:05.858 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 13/20 2020-03-28T06:16:05.928 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 15/20 2020-03-28T06:16:06.241 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-mako-0.8.1-2.el7.noarch 16/20 2020-03-28T06:16:06.243 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mon-14.2.8-334.gcd131a1eb0.el7.x86_64 14/20 2020-03-28T06:16:06.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 17/20 2020-03-28T06:16:06.646 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 15/20 2020-03-28T06:16:06.951 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-mako-0.8.1-2.el7.noarch 16/20 2020-03-28T06:16:07.328 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 18/20 2020-03-28T06:16:07.412 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 17/20 2020-03-28T06:16:07.729 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 19/20 2020-03-28T06:16:07.991 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 20/20 2020-03-28T06:16:08.120 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 1/20 2020-03-28T06:16:08.130 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 18/20 2020-03-28T06:16:08.229 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 2/20 2020-03-28T06:16:08.557 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 19/20 2020-03-28T06:16:08.567 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 3/20 2020-03-28T06:16:08.688 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mon-14.2.8-334.gcd131a1eb0.el7.x86_64 4/20 2020-03-28T06:16:08.797 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 5/20 2020-03-28T06:16:08.835 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 20/20 2020-03-28T06:16:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 6/20 2020-03-28T06:16:08.978 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 1/20 2020-03-28T06:16:09.007 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 7/20 2020-03-28T06:16:09.098 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 2/20 2020-03-28T06:16:09.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 8/20 2020-03-28T06:16:09.207 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 3/20 2020-03-28T06:16:09.233 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 9/20 2020-03-28T06:16:09.325 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 10/20 2020-03-28T06:16:09.333 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mon-14.2.8-334.gcd131a1eb0.el7.x86_64 4/20 2020-03-28T06:16:09.417 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 11/20 2020-03-28T06:16:09.433 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 5/20 2020-03-28T06:16:09.559 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 12/20 2020-03-28T06:16:09.575 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 6/20 2020-03-28T06:16:09.651 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 13/20 2020-03-28T06:16:09.692 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 7/20 2020-03-28T06:16:09.752 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 14/20 2020-03-28T06:16:09.793 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 8/20 2020-03-28T06:16:09.852 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-osd-14.2.8-334.gcd131a1eb0.el7.x86_64 15/20 2020-03-28T06:16:09.910 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 9/20 2020-03-28T06:16:09.953 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 16/20 2020-03-28T06:16:10.036 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 10/20 2020-03-28T06:16:10.062 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 17/20 2020-03-28T06:16:10.136 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 11/20 2020-03-28T06:16:10.171 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 18/20 2020-03-28T06:16:10.245 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 12/20 2020-03-28T06:16:10.271 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mds-14.2.8-334.gcd131a1eb0.el7.x86_64 19/20 2020-03-28T06:16:10.337 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 13/20 2020-03-28T06:16:10.489 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 14/20 2020-03-28T06:16:10.605 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-osd-14.2.8-334.gcd131a1eb0.el7.x86_64 15/20 2020-03-28T06:16:10.705 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 16/20 2020-03-28T06:16:10.758 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 20/20 2020-03-28T06:16:10.758 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:10.758 INFO:teuthology.orchestra.run.smithi156.stdout: ceph.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:10.758 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:16:10.758 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2020-03-28T06:16:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: python-beaker.noarch 0:1.5.4-10.el7 2020-03-28T06:16:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2020-03-28T06:16:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: python-mako.noarch 0:0.8.1-2.el7 2020-03-28T06:16:10.760 INFO:teuthology.orchestra.run.smithi156.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2020-03-28T06:16:10.760 INFO:teuthology.orchestra.run.smithi156.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2020-03-28T06:16:10.760 INFO:teuthology.orchestra.run.smithi156.stdout: python-pecan.noarch 0:0.4.5-2.el7 2020-03-28T06:16:10.760 INFO:teuthology.orchestra.run.smithi156.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2020-03-28T06:16:10.760 INFO:teuthology.orchestra.run.smithi156.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2020-03-28T06:16:10.760 INFO:teuthology.orchestra.run.smithi156.stdout: python-tempita.noarch 0:0.5.1-6.el7 2020-03-28T06:16:10.760 INFO:teuthology.orchestra.run.smithi156.stdout: python-webob.noarch 0:1.2.3-7.el7 2020-03-28T06:16:10.761 INFO:teuthology.orchestra.run.smithi156.stdout: python-webtest.noarch 0:1.3.4-6.el7 2020-03-28T06:16:10.761 INFO:teuthology.orchestra.run.smithi156.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2020-03-28T06:16:10.761 INFO:teuthology.orchestra.run.smithi156.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2020-03-28T06:16:10.761 INFO:teuthology.orchestra.run.smithi156.stdout: python2-six.noarch 0:1.9.0-0.el7 2020-03-28T06:16:10.761 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:10.761 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:10.823 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 17/20 2020-03-28T06:16:10.879 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr 2020-03-28T06:16:10.932 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 18/20 2020-03-28T06:16:11.074 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mds-14.2.8-334.gcd131a1eb0.el7.x86_64 19/20 2020-03-28T06:16:11.113 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:11.154 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:11.155 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:11.156 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:11.157 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:11.157 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:11.569 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-14.2.8-334.gcd131a1eb0.el7.x86_64 20/20 2020-03-28T06:16:11.569 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:11.569 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:11.569 INFO:teuthology.orchestra.run.smithi168.stdout: ceph.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:11.569 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:11.569 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:16:11.569 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mds.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:11.570 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:11.570 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mon.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:11.570 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-osd.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:11.570 INFO:teuthology.orchestra.run.smithi168.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2020-03-28T06:16:11.570 INFO:teuthology.orchestra.run.smithi168.stdout: python-beaker.noarch 0:1.5.4-10.el7 2020-03-28T06:16:11.570 INFO:teuthology.orchestra.run.smithi168.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2020-03-28T06:16:11.570 INFO:teuthology.orchestra.run.smithi168.stdout: python-mako.noarch 0:0.8.1-2.el7 2020-03-28T06:16:11.571 INFO:teuthology.orchestra.run.smithi168.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2020-03-28T06:16:11.571 INFO:teuthology.orchestra.run.smithi168.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2020-03-28T06:16:11.571 INFO:teuthology.orchestra.run.smithi168.stdout: python-pecan.noarch 0:0.4.5-2.el7 2020-03-28T06:16:11.571 INFO:teuthology.orchestra.run.smithi168.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2020-03-28T06:16:11.571 INFO:teuthology.orchestra.run.smithi168.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2020-03-28T06:16:11.571 INFO:teuthology.orchestra.run.smithi168.stdout: python-tempita.noarch 0:0.5.1-6.el7 2020-03-28T06:16:11.572 INFO:teuthology.orchestra.run.smithi168.stdout: python-webob.noarch 0:1.2.3-7.el7 2020-03-28T06:16:11.572 INFO:teuthology.orchestra.run.smithi168.stdout: python-webtest.noarch 0:1.3.4-6.el7 2020-03-28T06:16:11.572 INFO:teuthology.orchestra.run.smithi168.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2020-03-28T06:16:11.572 INFO:teuthology.orchestra.run.smithi168.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2020-03-28T06:16:11.572 INFO:teuthology.orchestra.run.smithi168.stdout: python2-six.noarch 0:1.9.0-0.el7 2020-03-28T06:16:11.572 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:11.572 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:11.694 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-mgr 2020-03-28T06:16:11.821 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:11.999 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:12.044 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:12.045 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:12.046 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:12.046 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:12.047 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:12.154 INFO:teuthology.orchestra.run.smithi156.stdout:Package 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:16:12.155 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do 2020-03-28T06:16:12.218 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-dashboard 2020-03-28T06:16:12.415 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:12.449 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:12.450 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:12.451 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:12.451 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:12.451 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:12.722 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:13.054 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:13.065 INFO:teuthology.orchestra.run.smithi168.stdout:Package 2:ceph-mgr-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:16:13.065 INFO:teuthology.orchestra.run.smithi168.stdout:Nothing to do 2020-03-28T06:16:13.127 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-mgr-dashboard 2020-03-28T06:16:13.284 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:13.318 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:13.319 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:13.319 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:13.320 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:13.320 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:13.387 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:13.387 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:13.388 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:13.390 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:13.421 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:13.422 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:13.422 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:13.422 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2020-03-28T06:16:13.423 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2020-03-28T06:16:13.424 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:13.424 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2020-03-28T06:16:13.799 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:13.922 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:14.030 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:14.031 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:16:14.032 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:14.032 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:14.032 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2020-03-28T06:16:14.033 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:14.033 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:16:14.033 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 4.0 M 2020-03-28T06:16:14.033 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:16:14.033 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 18 k 2020-03-28T06:16:14.034 INFO:teuthology.orchestra.run.smithi156.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2020-03-28T06:16:14.034 INFO:teuthology.orchestra.run.smithi156.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2020-03-28T06:16:14.035 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:14.035 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:16:14.035 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:14.035 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+3 Dependent packages) 2020-03-28T06:16:14.036 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:14.036 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 4.6 M 2020-03-28T06:16:14.036 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 20 M 2020-03-28T06:16:14.036 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:16:14.254 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:16:14.255 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:14.255 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:14.258 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:14.288 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:14.289 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:14.289 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:14.290 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2020-03-28T06:16:14.290 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2020-03-28T06:16:14.291 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:14.291 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2020-03-28T06:16:14.659 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:14.888 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:14.888 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:16:14.889 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:14.889 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:14.889 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository Size 2020-03-28T06:16:14.889 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:14.890 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:16:14.890 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-dashboard noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 4.0 M 2020-03-28T06:16:14.890 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:16:14.890 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-grafana-dashboards noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 18 k 2020-03-28T06:16:14.890 INFO:teuthology.orchestra.run.smithi168.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2020-03-28T06:16:14.890 INFO:teuthology.orchestra.run.smithi168.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2020-03-28T06:16:14.890 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:14.891 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:16:14.891 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:14.891 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+3 Dependent packages) 2020-03-28T06:16:14.891 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:14.891 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 4.6 M 2020-03-28T06:16:14.891 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 20 M 2020-03-28T06:16:14.891 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:16:15.893 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:15.893 INFO:teuthology.orchestra.run.smithi156.stdout:Total 2.5 MB/s | 4.6 MB 00:01 2020-03-28T06:16:15.898 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:16:15.906 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:16:16.081 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:16:16.090 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:16:16.718 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-grafana-dashboards-14.2.8-334.gcd131a1eb0.el7.noarc 1/4 2020-03-28T06:16:17.106 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:17.106 INFO:teuthology.orchestra.run.smithi168.stdout:Total 2.1 MB/s | 4.6 MB 00:02 2020-03-28T06:16:17.111 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:16:17.118 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:16:17.140 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 2/4 2020-03-28T06:16:17.325 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:16:17.325 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:16:18.032 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-grafana-dashboards-14.2.8-334.gcd131a1eb0.el7.noarc 1/4 2020-03-28T06:16:18.308 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-routes-1.13-2.el7.noarch 3/4 2020-03-28T06:16:18.450 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 2/4 2020-03-28T06:16:18.876 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 4/4 2020-03-28T06:16:18.986 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 1/4 2020-03-28T06:16:19.094 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-routes-1.13-2.el7.noarch 2/4 2020-03-28T06:16:19.234 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 3/4 2020-03-28T06:16:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-routes-1.13-2.el7.noarch 3/4 2020-03-28T06:16:19.713 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.8-334.gcd131a1eb0.el7.noarc 4/4 2020-03-28T06:16:19.713 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:19.713 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:19.713 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:19.713 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:19.713 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:16:19.714 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:19.714 INFO:teuthology.orchestra.run.smithi156.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2020-03-28T06:16:19.714 INFO:teuthology.orchestra.run.smithi156.stdout: python-routes.noarch 0:1.13-2.el7 2020-03-28T06:16:19.714 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:19.714 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:19.823 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-diskprediction-cloud 2020-03-28T06:16:19.972 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 4/4 2020-03-28T06:16:20.016 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:20.055 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:20.056 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:20.056 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:20.056 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:20.057 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:20.089 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.8-334.gcd131a1eb0.el7.noarch 1/4 2020-03-28T06:16:20.222 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-routes-1.13-2.el7.noarch 2/4 2020-03-28T06:16:20.323 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 3/4 2020-03-28T06:16:20.707 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:20.846 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.8-334.gcd131a1eb0.el7.noarc 4/4 2020-03-28T06:16:20.846 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:20.847 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:20.847 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-dashboard.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:20.847 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:20.847 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:16:20.847 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-grafana-dashboards.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:20.847 INFO:teuthology.orchestra.run.smithi168.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2020-03-28T06:16:20.847 INFO:teuthology.orchestra.run.smithi168.stdout: python-routes.noarch 0:1.13-2.el7 2020-03-28T06:16:20.848 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:20.848 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:20.966 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-mgr-diskprediction-cloud 2020-03-28T06:16:21.041 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:21.042 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:21.042 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:21.190 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:21.228 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:21.229 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:21.229 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:21.229 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:21.230 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:21.408 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:21.636 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:21.637 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:16:21.637 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:21.638 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:21.638 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2020-03-28T06:16:21.638 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:21.638 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:16:21.638 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-cloud 2020-03-28T06:16:21.638 INFO:teuthology.orchestra.run.smithi156.stdout: noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 83 k 2020-03-28T06:16:21.638 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:21.639 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:16:21.639 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:21.639 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2020-03-28T06:16:21.639 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:21.639 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 83 k 2020-03-28T06:16:21.639 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 412 k 2020-03-28T06:16:21.639 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:16:21.868 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:22.045 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:16:22.049 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:16:22.104 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:16:22.105 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:16:22.206 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:16:22.206 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:22.206 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:22.578 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:22.810 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.8-334.gcd131a1eb0.el7 1/1 2020-03-28T06:16:22.818 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:22.818 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:16:22.819 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:22.819 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:22.819 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository Size 2020-03-28T06:16:22.820 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:22.820 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:16:22.820 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-diskprediction-cloud 2020-03-28T06:16:22.820 INFO:teuthology.orchestra.run.smithi168.stdout: noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 83 k 2020-03-28T06:16:22.821 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:22.821 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:16:22.821 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:22.821 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package 2020-03-28T06:16:22.822 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:22.822 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 83 k 2020-03-28T06:16:22.822 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 412 k 2020-03-28T06:16:22.822 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:16:23.266 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.8-334.gcd131a1eb0.el7 1/1 2020-03-28T06:16:23.266 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:23.266 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:23.267 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:23.267 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:23.267 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:23.275 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:16:23.278 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:16:23.327 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:16:23.328 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:16:23.383 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-28T06:16:23.616 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:23.658 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:23.659 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:23.659 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:23.659 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:23.660 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:23.956 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.8-334.gcd131a1eb0.el7 1/1 2020-03-28T06:16:24.337 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:24.487 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.8-334.gcd131a1eb0.el7 1/1 2020-03-28T06:16:24.487 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:24.488 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:24.488 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:24.488 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:24.488 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:24.612 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-28T06:16:24.677 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:24.677 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:24.677 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:24.680 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:24.710 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:24.711 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2020-03-28T06:16:24.717 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2020-03-28T06:16:24.718 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2020-03-28T06:16:24.721 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2020-03-28T06:16:24.721 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:24.721 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2020-03-28T06:16:24.722 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2020-03-28T06:16:24.723 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2020-03-28T06:16:24.724 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2020-03-28T06:16:24.725 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:24.725 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2020-03-28T06:16:24.884 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:24.930 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:24.931 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:24.931 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:24.932 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:24.932 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:25.090 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:25.317 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:25.317 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:16:25.318 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:25.318 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:25.318 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout: noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 1.1 M 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2020-03-28T06:16:25.319 INFO:teuthology.orchestra.run.smithi156.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2020-03-28T06:16:25.320 INFO:teuthology.orchestra.run.smithi156.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2020-03-28T06:16:25.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:25.320 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:16:25.320 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:25.320 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+4 Dependent packages) 2020-03-28T06:16:25.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:25.320 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 12 M 2020-03-28T06:16:25.321 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 66 M 2020-03-28T06:16:25.321 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:16:25.599 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:25.948 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:16:25.949 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:25.949 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:25.951 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:25.982 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:25.982 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2020-03-28T06:16:25.989 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2020-03-28T06:16:25.990 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2020-03-28T06:16:25.992 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2020-03-28T06:16:25.993 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:25.993 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2020-03-28T06:16:25.993 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2020-03-28T06:16:25.994 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2020-03-28T06:16:25.996 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2020-03-28T06:16:25.996 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:25.997 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2020-03-28T06:16:26.302 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:26.303 INFO:teuthology.orchestra.run.smithi156.stdout:Total 12 MB/s | 12 MB 00:00 2020-03-28T06:16:26.311 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:16:26.334 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:16:26.360 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:26.428 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:16:26.428 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:16:26.587 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:26.588 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:16:26.589 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:26.589 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:26.589 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository Size 2020-03-28T06:16:26.589 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:26.589 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:16:26.589 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-diskprediction-local 2020-03-28T06:16:26.589 INFO:teuthology.orchestra.run.smithi168.stdout: noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 1.1 M 2020-03-28T06:16:26.590 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:16:26.590 INFO:teuthology.orchestra.run.smithi168.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2020-03-28T06:16:26.590 INFO:teuthology.orchestra.run.smithi168.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2020-03-28T06:16:26.590 INFO:teuthology.orchestra.run.smithi168.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2020-03-28T06:16:26.590 INFO:teuthology.orchestra.run.smithi168.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2020-03-28T06:16:26.590 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:26.590 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:16:26.591 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:26.591 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+4 Dependent packages) 2020-03-28T06:16:26.591 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:26.591 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 12 M 2020-03-28T06:16:26.591 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 66 M 2020-03-28T06:16:26.591 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:16:27.342 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2020-03-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:27.785 INFO:teuthology.orchestra.run.smithi168.stdout:Total 9.7 MB/s | 12 MB 00:01 2020-03-28T06:16:27.795 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:16:27.815 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2020-03-28T06:16:27.820 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:16:27.938 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:16:27.939 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:16:28.861 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2020-03-28T06:16:29.350 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2020-03-28T06:16:29.560 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2020-03-28T06:16:30.384 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2020-03-28T06:16:30.691 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.8-334.gcd131a1eb0.el7 5/5 2020-03-28T06:16:30.789 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.8-334.gcd131a1eb0.el7 1/5 2020-03-28T06:16:30.890 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 2/5 2020-03-28T06:16:31.024 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 3/5 2020-03-28T06:16:31.105 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2020-03-28T06:16:31.142 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 4/5 2020-03-28T06:16:31.626 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2020-03-28T06:16:31.626 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:31.626 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:31.627 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:31.627 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:31.627 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:16:31.627 INFO:teuthology.orchestra.run.smithi156.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2020-03-28T06:16:31.627 INFO:teuthology.orchestra.run.smithi156.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2020-03-28T06:16:31.627 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:31.627 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:31.749 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-rook 2020-03-28T06:16:31.945 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2020-03-28T06:16:31.971 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:32.015 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:32.016 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:32.017 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:32.017 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:32.017 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:32.662 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.8-334.gcd131a1eb0.el7 5/5 2020-03-28T06:16:32.695 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:32.794 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.8-334.gcd131a1eb0.el7 1/5 2020-03-28T06:16:32.911 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 2/5 2020-03-28T06:16:33.035 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:33.036 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:33.036 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:33.036 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 3/5 2020-03-28T06:16:33.037 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:33.068 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:33.068 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-2.el7 will be installed 2020-03-28T06:16:33.071 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:33.072 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:33.073 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:33.074 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:33.075 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:33.075 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2020-03-28T06:16:33.075 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2020-03-28T06:16:33.075 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2020-03-28T06:16:33.076 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2020-03-28T06:16:33.077 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2020-03-28T06:16:33.078 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2020-03-28T06:16:33.079 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2020-03-28T06:16:33.079 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2020-03-28T06:16:33.080 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:33.080 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2020-03-28T06:16:33.080 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2020-03-28T06:16:33.081 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2020-03-28T06:16:33.082 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2020-03-28T06:16:33.082 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2020-03-28T06:16:33.082 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:33.082 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2020-03-28T06:16:33.145 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 4/5 2020-03-28T06:16:33.450 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:33.659 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2020-03-28T06:16:33.659 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:33.660 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:33.660 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:33.660 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:33.660 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:16:33.661 INFO:teuthology.orchestra.run.smithi168.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2020-03-28T06:16:33.661 INFO:teuthology.orchestra.run.smithi168.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2020-03-28T06:16:33.662 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:33.662 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:33.683 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:33.683 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:16:33.685 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:33.685 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:33.685 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2020-03-28T06:16:33.685 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:33.686 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:16:33.686 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 21 k 2020-03-28T06:16:33.686 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:16:33.686 INFO:teuthology.orchestra.run.smithi156.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2020-03-28T06:16:33.686 INFO:teuthology.orchestra.run.smithi156.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2020-03-28T06:16:33.686 INFO:teuthology.orchestra.run.smithi156.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2020-03-28T06:16:33.686 INFO:teuthology.orchestra.run.smithi156.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2020-03-28T06:16:33.687 INFO:teuthology.orchestra.run.smithi156.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2020-03-28T06:16:33.687 INFO:teuthology.orchestra.run.smithi156.stdout: python2-kubernetes noarch 10.0.1-2.el7 epel 1.1 M 2020-03-28T06:16:33.687 INFO:teuthology.orchestra.run.smithi156.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2020-03-28T06:16:33.687 INFO:teuthology.orchestra.run.smithi156.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2020-03-28T06:16:33.687 INFO:teuthology.orchestra.run.smithi156.stdout: python2-requests-oauthlib 2020-03-28T06:16:33.687 INFO:teuthology.orchestra.run.smithi156.stdout: noarch 0.8.0-5.el7 base 35 k 2020-03-28T06:16:33.687 INFO:teuthology.orchestra.run.smithi156.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2020-03-28T06:16:33.688 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:33.688 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:16:33.688 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:33.688 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+10 Dependent packages) 2020-03-28T06:16:33.688 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:33.689 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 1.6 M 2020-03-28T06:16:33.689 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 24 M 2020-03-28T06:16:33.689 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:16:33.788 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-mgr-rook 2020-03-28T06:16:34.011 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:34.051 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:34.052 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:34.052 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:34.052 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:34.053 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:34.087 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:34.087 INFO:teuthology.orchestra.run.smithi156.stdout:Total 4.1 MB/s | 1.6 MB 00:00 2020-03-28T06:16:34.099 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:16:34.109 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:16:34.216 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:16:34.216 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:16:34.753 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:35.017 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 1/11 2020-03-28T06:16:35.090 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:16:35.090 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:35.091 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:35.092 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:35.123 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:35.123 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-2.el7 will be installed 2020-03-28T06:16:35.125 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:35.126 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:35.127 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:35.128 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-2.el7.noarch 2020-03-28T06:16:35.129 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:35.130 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2020-03-28T06:16:35.130 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2020-03-28T06:16:35.130 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2020-03-28T06:16:35.131 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2020-03-28T06:16:35.131 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2020-03-28T06:16:35.132 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2020-03-28T06:16:35.133 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2020-03-28T06:16:35.133 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2020-03-28T06:16:35.134 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:35.134 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2020-03-28T06:16:35.135 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2020-03-28T06:16:35.135 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2020-03-28T06:16:35.136 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2020-03-28T06:16:35.136 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2020-03-28T06:16:35.136 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:35.137 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2020-03-28T06:16:35.369 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 2/11 2020-03-28T06:16:35.505 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:35.737 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:35.738 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:16:35.739 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:35.739 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:35.739 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository Size 2020-03-28T06:16:35.739 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:35.740 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:16:35.740 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-rook noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 21 k 2020-03-28T06:16:35.740 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:16:35.740 INFO:teuthology.orchestra.run.smithi168.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2020-03-28T06:16:35.741 INFO:teuthology.orchestra.run.smithi168.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2020-03-28T06:16:35.741 INFO:teuthology.orchestra.run.smithi168.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2020-03-28T06:16:35.741 INFO:teuthology.orchestra.run.smithi168.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2020-03-28T06:16:35.741 INFO:teuthology.orchestra.run.smithi168.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2020-03-28T06:16:35.741 INFO:teuthology.orchestra.run.smithi168.stdout: python2-kubernetes noarch 10.0.1-2.el7 epel 1.1 M 2020-03-28T06:16:35.741 INFO:teuthology.orchestra.run.smithi168.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2020-03-28T06:16:35.742 INFO:teuthology.orchestra.run.smithi168.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2020-03-28T06:16:35.742 INFO:teuthology.orchestra.run.smithi168.stdout: python2-requests-oauthlib 2020-03-28T06:16:35.742 INFO:teuthology.orchestra.run.smithi168.stdout: noarch 0.8.0-5.el7 base 35 k 2020-03-28T06:16:35.742 INFO:teuthology.orchestra.run.smithi168.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2020-03-28T06:16:35.742 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:35.742 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:16:35.743 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:35.743 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+10 Dependent packages) 2020-03-28T06:16:35.743 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:35.744 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 3/11 2020-03-28T06:16:35.744 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 1.6 M 2020-03-28T06:16:35.744 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 24 M 2020-03-28T06:16:35.745 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:16:36.016 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 4/11 2020-03-28T06:16:36.151 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:36.151 INFO:teuthology.orchestra.run.smithi168.stdout:Total 4.0 MB/s | 1.6 MB 00:00 2020-03-28T06:16:36.162 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:16:36.172 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:16:36.266 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:16:36.266 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:16:36.332 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2020-03-28T06:16:36.704 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2020-03-28T06:16:37.035 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 1/11 2020-03-28T06:16:37.087 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2020-03-28T06:16:37.391 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2020-03-28T06:16:37.566 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 2/11 2020-03-28T06:16:37.928 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 3/11 2020-03-28T06:16:38.223 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 4/11 2020-03-28T06:16:38.537 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2020-03-28T06:16:38.677 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2020-03-28T06:16:38.893 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2020-03-28T06:16:39.252 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2020-03-28T06:16:39.337 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-kubernetes-10.0.1-2.el7.noarch 10/11 2020-03-28T06:16:39.614 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mgr-rook-14.2.8-334.gcd131a1eb0.el7.noarch 11/11 2020-03-28T06:16:39.615 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2020-03-28T06:16:39.721 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2020-03-28T06:16:39.830 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 2/11 2020-03-28T06:16:39.930 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mgr-rook-14.2.8-334.gcd131a1eb0.el7.noarch 3/11 2020-03-28T06:16:40.031 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 4/11 2020-03-28T06:16:40.165 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 5/11 2020-03-28T06:16:40.273 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 6/11 2020-03-28T06:16:40.374 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 7/11 2020-03-28T06:16:40.491 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 8/11 2020-03-28T06:16:40.592 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 9/11 2020-03-28T06:16:40.692 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 10/11 2020-03-28T06:16:40.943 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2020-03-28T06:16:41.187 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-kubernetes-10.0.1-2.el7.noarch 11/11 2020-03-28T06:16:41.187 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:41.187 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:41.188 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:41.188 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:41.188 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:16:41.188 INFO:teuthology.orchestra.run.smithi156.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2020-03-28T06:16:41.188 INFO:teuthology.orchestra.run.smithi156.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2020-03-28T06:16:41.189 INFO:teuthology.orchestra.run.smithi156.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2020-03-28T06:16:41.189 INFO:teuthology.orchestra.run.smithi156.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2020-03-28T06:16:41.189 INFO:teuthology.orchestra.run.smithi156.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2020-03-28T06:16:41.189 INFO:teuthology.orchestra.run.smithi156.stdout: python2-kubernetes.noarch 0:10.0.1-2.el7 2020-03-28T06:16:41.189 INFO:teuthology.orchestra.run.smithi156.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2020-03-28T06:16:41.190 INFO:teuthology.orchestra.run.smithi156.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2020-03-28T06:16:41.190 INFO:teuthology.orchestra.run.smithi156.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2020-03-28T06:16:41.190 INFO:teuthology.orchestra.run.smithi156.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2020-03-28T06:16:41.190 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:41.190 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:41.315 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-kubernetes-10.0.1-2.el7.noarch 10/11 2020-03-28T06:16:41.318 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-ssh 2020-03-28T06:16:41.531 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:41.568 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:41.569 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:41.570 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:41.570 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:41.570 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:41.606 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mgr-rook-14.2.8-334.gcd131a1eb0.el7.noarch 11/11 2020-03-28T06:16:41.728 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2020-03-28T06:16:41.878 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 2/11 2020-03-28T06:16:42.138 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mgr-rook-14.2.8-334.gcd131a1eb0.el7.noarch 3/11 2020-03-28T06:16:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:42.247 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 4/11 2020-03-28T06:16:42.381 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 5/11 2020-03-28T06:16:42.481 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 6/11 2020-03-28T06:16:42.549 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:42.553 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:42.553 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:42.553 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:42.565 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 7/11 2020-03-28T06:16:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:42.582 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2020-03-28T06:16:42.583 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2020-03-28T06:16:42.584 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:42.585 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2020-03-28T06:16:42.585 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2020-03-28T06:16:42.585 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:42.585 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python2-apipkg.noarch 0:1.5-4.el7 will be installed 2020-03-28T06:16:42.683 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 8/11 2020-03-28T06:16:42.817 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 9/11 2020-03-28T06:16:42.933 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 10/11 2020-03-28T06:16:42.957 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:43.189 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:43.189 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:16:43.190 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:43.190 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:43.190 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2020-03-28T06:16:43.191 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:43.191 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:16:43.191 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-ssh noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 21 k 2020-03-28T06:16:43.191 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:16:43.191 INFO:teuthology.orchestra.run.smithi156.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2020-03-28T06:16:43.191 INFO:teuthology.orchestra.run.smithi156.stdout: python2-apipkg noarch 1.5-4.el7 epel 17 k 2020-03-28T06:16:43.191 INFO:teuthology.orchestra.run.smithi156.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2020-03-28T06:16:43.192 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:43.192 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:16:43.192 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:43.192 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+3 Dependent packages) 2020-03-28T06:16:43.194 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:43.194 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 340 k 2020-03-28T06:16:43.194 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 1.1 M 2020-03-28T06:16:43.194 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:16:43.462 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-kubernetes-10.0.1-2.el7.noarch 11/11 2020-03-28T06:16:43.462 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:43.462 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:43.462 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-rook.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:43.463 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:43.463 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:16:43.463 INFO:teuthology.orchestra.run.smithi168.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2020-03-28T06:16:43.463 INFO:teuthology.orchestra.run.smithi168.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2020-03-28T06:16:43.463 INFO:teuthology.orchestra.run.smithi168.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2020-03-28T06:16:43.463 INFO:teuthology.orchestra.run.smithi168.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2020-03-28T06:16:43.464 INFO:teuthology.orchestra.run.smithi168.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2020-03-28T06:16:43.464 INFO:teuthology.orchestra.run.smithi168.stdout: python2-kubernetes.noarch 0:10.0.1-2.el7 2020-03-28T06:16:43.464 INFO:teuthology.orchestra.run.smithi168.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2020-03-28T06:16:43.464 INFO:teuthology.orchestra.run.smithi168.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2020-03-28T06:16:43.464 INFO:teuthology.orchestra.run.smithi168.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2020-03-28T06:16:43.464 INFO:teuthology.orchestra.run.smithi168.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2020-03-28T06:16:43.464 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:43.465 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:43.583 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-mgr-ssh 2020-03-28T06:16:43.769 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:43.770 INFO:teuthology.orchestra.run.smithi156.stdout:Total 587 kB/s | 340 kB 00:00 2020-03-28T06:16:43.774 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:16:43.781 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:16:43.822 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:16:43.823 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:16:43.838 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:43.881 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:43.882 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:43.883 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:43.883 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:43.883 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:44.492 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-apipkg-1.5-4.el7.noarch 1/4 2020-03-28T06:16:44.553 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:44.887 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 2/4 2020-03-28T06:16:44.893 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:16:44.894 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:44.894 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:44.896 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.8-334.gcd131a1eb0.el7.noarch 2020-03-28T06:16:44.926 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:44.927 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2020-03-28T06:16:44.927 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2020-03-28T06:16:44.928 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:44.929 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2020-03-28T06:16:44.929 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2020-03-28T06:16:44.929 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:44.929 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python2-apipkg.noarch 0:1.5-4.el7 will be installed 2020-03-28T06:16:45.260 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 3/4 2020-03-28T06:16:45.300 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:45.530 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:45.530 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:16:45.530 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:45.531 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:45.531 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository Size 2020-03-28T06:16:45.531 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:45.531 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:16:45.531 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-ssh noarch 2:14.2.8-334.gcd131a1eb0.el7 ceph-noarch 21 k 2020-03-28T06:16:45.531 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:16:45.531 INFO:teuthology.orchestra.run.smithi168.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2020-03-28T06:16:45.532 INFO:teuthology.orchestra.run.smithi168.stdout: python2-apipkg noarch 1.5-4.el7 epel 17 k 2020-03-28T06:16:45.532 INFO:teuthology.orchestra.run.smithi168.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2020-03-28T06:16:45.532 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:45.532 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:16:45.532 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:45.532 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+3 Dependent packages) 2020-03-28T06:16:45.532 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:45.533 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 340 k 2020-03-28T06:16:45.533 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 1.1 M 2020-03-28T06:16:45.533 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:16:45.566 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-mgr-ssh-14.2.8-334.gcd131a1eb0.el7.noarch 4/4 2020-03-28T06:16:45.653 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-mgr-ssh-14.2.8-334.gcd131a1eb0.el7.noarch 1/4 2020-03-28T06:16:45.745 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 2/4 2020-03-28T06:16:45.845 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 3/4 2020-03-28T06:16:46.016 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:46.016 INFO:teuthology.orchestra.run.smithi168.stdout:Total 702 kB/s | 340 kB 00:00 2020-03-28T06:16:46.020 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:16:46.026 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:16:46.065 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:16:46.066 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:16:46.322 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python2-apipkg-1.5-4.el7.noarch 4/4 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-ssh.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout: python-execnet.noarch 0:1.7.1-3.el7 python2-apipkg.noarch 0:1.5-4.el7 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2020-03-28T06:16:46.323 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:46.324 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:46.486 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-fuse 2020-03-28T06:16:46.708 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:46.752 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:46.753 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:46.753 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:46.753 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:46.754 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:46.806 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-apipkg-1.5-4.el7.noarch 1/4 2020-03-28T06:16:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:47.436 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 2/4 2020-03-28T06:16:47.763 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:47.764 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:47.764 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package ceph-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:47.817 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 3/4 2020-03-28T06:16:48.106 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-mgr-ssh-14.2.8-334.gcd131a1eb0.el7.noarch 4/4 2020-03-28T06:16:48.142 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:48.210 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-mgr-ssh-14.2.8-334.gcd131a1eb0.el7.noarch 1/4 2020-03-28T06:16:48.343 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 2/4 2020-03-28T06:16:48.373 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:48.373 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:16:48.374 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:48.374 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:48.374 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository 2020-03-28T06:16:48.374 INFO:teuthology.orchestra.run.smithi156.stdout: Size 2020-03-28T06:16:48.375 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:48.375 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:16:48.375 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 581 k 2020-03-28T06:16:48.375 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:48.375 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:16:48.375 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:48.375 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2020-03-28T06:16:48.376 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:48.376 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 581 k 2020-03-28T06:16:48.376 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 1.6 M 2020-03-28T06:16:48.376 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:16:48.452 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 3/4 2020-03-28T06:16:48.979 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : python2-apipkg-1.5-4.el7.noarch 4/4 2020-03-28T06:16:48.979 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:48.979 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:48.979 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-ssh.noarch 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:48.980 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:48.980 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:16:48.980 INFO:teuthology.orchestra.run.smithi168.stdout: python-execnet.noarch 0:1.7.1-3.el7 python2-apipkg.noarch 0:1.5-4.el7 2020-03-28T06:16:48.980 INFO:teuthology.orchestra.run.smithi168.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2020-03-28T06:16:48.980 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:48.980 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:49.117 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install ceph-fuse 2020-03-28T06:16:49.171 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:16:49.197 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:16:49.199 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:16:49.199 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:16:49.342 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:49.385 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:49.386 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:49.386 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:49.387 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:49.387 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:49.870 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:ceph-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:16:50.058 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:50.326 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:ceph-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:16:50.327 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:50.327 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:50.327 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:50.327 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:50.327 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:50.396 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:16:50.397 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:50.397 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package ceph-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:50.444 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs2 2020-03-28T06:16:50.657 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:50.695 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:50.696 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:50.697 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:50.697 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:50.697 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:50.775 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:51.008 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:51.008 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:16:51.009 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:51.009 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:51.009 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository 2020-03-28T06:16:51.009 INFO:teuthology.orchestra.run.smithi168.stdout: Size 2020-03-28T06:16:51.009 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:51.009 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:16:51.009 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-fuse x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 581 k 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 581 k 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 1.6 M 2020-03-28T06:16:51.010 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:16:51.338 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:51.670 INFO:teuthology.orchestra.run.smithi156.stdout:Package 2:libcephfs2-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:16:51.670 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do 2020-03-28T06:16:51.736 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs-devel 2020-03-28T06:16:51.828 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:16:51.854 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:16:51.855 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:16:51.856 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:16:51.949 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:51.987 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:51.988 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:51.989 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:51.989 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:51.989 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:52.593 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:ceph-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:16:52.665 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:52.999 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:53.000 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:53.000 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package libcephfs-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:53.001 INFO:teuthology.orchestra.run.smithi156.stdout:--> Processing Dependency: librados-devel = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:libcephfs-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:16:53.032 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:53.032 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package librados-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:53.076 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:ceph-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:16:53.076 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:53.076 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:53.076 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:53.077 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:53.077 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:53.201 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install libcephfs2 2020-03-28T06:16:53.414 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:53.441 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:53.483 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:53.484 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:53.485 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:53.485 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:53.485 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:53.645 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:53.645 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:16:53.646 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:53.646 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:53.646 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository 2020-03-28T06:16:53.646 INFO:teuthology.orchestra.run.smithi156.stdout: Size 2020-03-28T06:16:53.647 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:53.647 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:16:53.647 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 17 k 2020-03-28T06:16:53.647 INFO:teuthology.orchestra.run.smithi156.stdout:Installing for dependencies: 2020-03-28T06:16:53.647 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 92 k 2020-03-28T06:16:53.647 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:53.647 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:16:53.648 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:16:53.648 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package (+1 Dependent package) 2020-03-28T06:16:53.648 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:53.648 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 109 k 2020-03-28T06:16:53.648 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 391 k 2020-03-28T06:16:53.648 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:16:54.142 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:54.359 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:54.360 INFO:teuthology.orchestra.run.smithi156.stdout:Total 153 kB/s | 109 kB 00:00 2020-03-28T06:16:54.362 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:16:54.386 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:16:54.388 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:16:54.389 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:16:54.479 INFO:teuthology.orchestra.run.smithi168.stdout:Package 2:libcephfs2-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:16:54.479 INFO:teuthology.orchestra.run.smithi168.stdout:Nothing to do 2020-03-28T06:16:54.543 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install libcephfs-devel 2020-03-28T06:16:54.705 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:54.739 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:54.740 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:54.741 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:54.741 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:54.742 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:54.956 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:librados-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 1/2 2020-03-28T06:16:55.259 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:libcephfs-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 2/2 2020-03-28T06:16:55.357 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:librados-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 1/2 2020-03-28T06:16:55.361 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:55.695 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:16:55.696 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:55.696 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package libcephfs-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:55.696 INFO:teuthology.orchestra.run.smithi168.stdout:--> Processing Dependency: librados-devel = 2:14.2.8-334.gcd131a1eb0.el7 for package: 2:libcephfs-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 2020-03-28T06:16:55.727 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:16:55.727 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package librados-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:55.833 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:libcephfs-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 2/2 2020-03-28T06:16:55.833 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:55.833 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:16:55.833 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:55.834 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:55.834 INFO:teuthology.orchestra.run.smithi156.stdout:Dependency Installed: 2020-03-28T06:16:55.834 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:55.834 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:16:55.834 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:16:55.958 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install librados2 2020-03-28T06:16:56.104 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:16:56.167 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:56.209 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:56.210 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:56.210 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:56.210 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:56.211 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:56.334 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:56.335 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:16:56.335 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:56.335 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:56.336 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository 2020-03-28T06:16:56.336 INFO:teuthology.orchestra.run.smithi168.stdout: Size 2020-03-28T06:16:56.336 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:56.336 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:16:56.336 INFO:teuthology.orchestra.run.smithi168.stdout: libcephfs-devel x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 17 k 2020-03-28T06:16:56.336 INFO:teuthology.orchestra.run.smithi168.stdout:Installing for dependencies: 2020-03-28T06:16:56.337 INFO:teuthology.orchestra.run.smithi168.stdout: librados-devel x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 92 k 2020-03-28T06:16:56.337 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:56.337 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:16:56.337 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:16:56.337 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package (+1 Dependent package) 2020-03-28T06:16:56.337 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:56.337 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 109 k 2020-03-28T06:16:56.338 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 391 k 2020-03-28T06:16:56.338 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:16:56.874 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:56.947 INFO:teuthology.orchestra.run.smithi168.stdout:-------------------------------------------------------------------------------- 2020-03-28T06:16:56.947 INFO:teuthology.orchestra.run.smithi168.stdout:Total 178 kB/s | 109 kB 00:00 2020-03-28T06:16:56.950 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:16:56.975 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:16:56.977 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:16:56.978 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:16:57.209 INFO:teuthology.orchestra.run.smithi156.stdout:Package 2:librados2-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:16:57.209 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do 2020-03-28T06:16:57.273 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install librbd1 2020-03-28T06:16:57.501 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:57.543 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:57.544 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:57.544 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:57.544 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:57.545 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:57.588 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:librados-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 1/2 2020-03-28T06:16:57.883 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:libcephfs-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 2/2 2020-03-28T06:16:57.989 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:librados-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 1/2 2020-03-28T06:16:58.202 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:58.464 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:libcephfs-devel-14.2.8-334.gcd131a1eb0.el7.x86_64 2/2 2020-03-28T06:16:58.464 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:58.465 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:16:58.465 INFO:teuthology.orchestra.run.smithi168.stdout: libcephfs-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:58.465 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:58.465 INFO:teuthology.orchestra.run.smithi168.stdout:Dependency Installed: 2020-03-28T06:16:58.465 INFO:teuthology.orchestra.run.smithi168.stdout: librados-devel.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:16:58.465 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:16:58.465 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:16:58.542 INFO:teuthology.orchestra.run.smithi156.stdout:Package 2:librbd1-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:16:58.542 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do 2020-03-28T06:16:58.593 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install librados2 2020-03-28T06:16:58.605 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install python-ceph 2020-03-28T06:16:58.759 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:58.793 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:58.794 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:58.794 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:58.794 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:58.795 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:16:58.803 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:16:58.841 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:16:58.842 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:16:58.842 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:16:58.842 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:16:58.843 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:16:59.412 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:59.503 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:16:59.745 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:16:59.838 INFO:teuthology.orchestra.run.smithi168.stdout:Package 2:librados2-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:16:59.838 INFO:teuthology.orchestra.run.smithi168.stdout:Nothing to do 2020-03-28T06:16:59.838 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:16:59.838 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package python-ceph-compat.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:16:59.901 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install librbd1 2020-03-28T06:17:00.071 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:17:00.105 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:17:00.106 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:17:00.107 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:17:00.107 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:17:00.107 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:17:00.116 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:17:00.348 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:00.348 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:17:00.349 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:00.349 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:17:00.349 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository 2020-03-28T06:17:00.350 INFO:teuthology.orchestra.run.smithi156.stdout: Size 2020-03-28T06:17:00.350 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:17:00.350 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:17:00.350 INFO:teuthology.orchestra.run.smithi156.stdout: python-ceph-compat x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 2.2 k 2020-03-28T06:17:00.350 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:00.350 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:17:00.350 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:17:00.351 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2020-03-28T06:17:00.351 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:00.351 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 2.2 k 2020-03-28T06:17:00.351 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 0 2020-03-28T06:17:00.351 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:17:00.708 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:17:00.799 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:17:00.801 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:17:00.803 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:17:00.804 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:17:01.041 INFO:teuthology.orchestra.run.smithi168.stdout:Package 2:librbd1-14.2.8-334.gcd131a1eb0.el7.x86_64 already installed and latest version 2020-03-28T06:17:01.041 INFO:teuthology.orchestra.run.smithi168.stdout:Nothing to do 2020-03-28T06:17:01.104 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install python-ceph 2020-03-28T06:17:01.262 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:17:01.296 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:17:01.297 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:17:01.298 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:17:01.298 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:17:01.298 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:17:01.329 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:python-ceph-compat-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:01.887 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:python-ceph-compat-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:01.887 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:01.887 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:17:01.887 INFO:teuthology.orchestra.run.smithi156.stdout: python-ceph-compat.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:01.888 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:01.888 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:17:01.905 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:17:02.013 INFO:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-fuse 2020-03-28T06:17:02.240 INFO:teuthology.orchestra.run.smithi156.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:17:02.250 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:17:02.250 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:17:02.250 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package python-ceph-compat.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:17:02.279 INFO:teuthology.orchestra.run.smithi156.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:17:02.280 INFO:teuthology.orchestra.run.smithi156.stdout: * base: mirror.trouble-free.net 2020-03-28T06:17:02.281 INFO:teuthology.orchestra.run.smithi156.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:17:02.281 INFO:teuthology.orchestra.run.smithi156.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:17:02.282 INFO:teuthology.orchestra.run.smithi156.stdout: * updates: mirror.datto.com 2020-03-28T06:17:02.633 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:17:02.869 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:02.870 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:17:02.870 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:02.870 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:17:02.871 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository 2020-03-28T06:17:02.871 INFO:teuthology.orchestra.run.smithi168.stdout: Size 2020-03-28T06:17:02.871 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:17:02.871 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:17:02.871 INFO:teuthology.orchestra.run.smithi168.stdout: python-ceph-compat x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 2.2 k 2020-03-28T06:17:02.871 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:02.872 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:17:02.872 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:17:02.872 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package 2020-03-28T06:17:02.872 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:02.872 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 2.2 k 2020-03-28T06:17:02.872 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 0 2020-03-28T06:17:02.872 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:17:02.945 INFO:teuthology.orchestra.run.smithi156.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:17:03.277 INFO:teuthology.orchestra.run.smithi156.stdout:Resolving Dependencies 2020-03-28T06:17:03.277 INFO:teuthology.orchestra.run.smithi156.stdout:--> Running transaction check 2020-03-28T06:17:03.278 INFO:teuthology.orchestra.run.smithi156.stdout:---> Package rbd-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:17:03.344 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:17:03.346 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:17:03.348 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:17:03.349 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:17:03.657 INFO:teuthology.orchestra.run.smithi156.stdout:--> Finished Dependency Resolution 2020-03-28T06:17:03.855 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:python-ceph-compat-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:03.889 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:03.889 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies Resolved 2020-03-28T06:17:03.890 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:03.891 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:17:03.891 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository 2020-03-28T06:17:03.891 INFO:teuthology.orchestra.run.smithi156.stdout: Size 2020-03-28T06:17:03.891 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:17:03.891 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2020-03-28T06:17:03.891 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 75 k 2020-03-28T06:17:03.891 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:03.892 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2020-03-28T06:17:03.892 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2020-03-28T06:17:03.892 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2020-03-28T06:17:03.892 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:03.892 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 75 k 2020-03-28T06:17:03.892 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 196 k 2020-03-28T06:17:03.892 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading packages: 2020-03-28T06:17:04.346 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:python-ceph-compat-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:04.346 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:04.346 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:17:04.346 INFO:teuthology.orchestra.run.smithi168.stdout: python-ceph-compat.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:04.347 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:04.347 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:17:04.471 INFO:teuthology.orchestra.run.smithi168:> sudo yum -y install rbd-fuse 2020-03-28T06:17:04.546 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2020-03-28T06:17:04.571 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2020-03-28T06:17:04.573 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded 2020-03-28T06:17:04.574 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2020-03-28T06:17:04.691 INFO:teuthology.orchestra.run.smithi168.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-03-28T06:17:04.731 INFO:teuthology.orchestra.run.smithi168.stdout:Loading mirror speeds from cached hostfile 2020-03-28T06:17:04.732 INFO:teuthology.orchestra.run.smithi168.stdout: * base: mirror.trouble-free.net 2020-03-28T06:17:04.732 INFO:teuthology.orchestra.run.smithi168.stdout: * epel: download-cc-rdu01.fedoraproject.org 2020-03-28T06:17:04.733 INFO:teuthology.orchestra.run.smithi168.stdout: * extras: packages.oit.ncsu.edu 2020-03-28T06:17:04.733 INFO:teuthology.orchestra.run.smithi168.stdout: * updates: mirror.datto.com 2020-03-28T06:17:05.119 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : 2:rbd-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:05.390 INFO:teuthology.orchestra.run.smithi168.stdout:7 packages excluded due to repository priority protections 2020-03-28T06:17:05.559 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : 2:rbd-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:05.560 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:05.560 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2020-03-28T06:17:05.560 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:05.560 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:17:05.560 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2020-03-28T06:17:05.688 DEBUG:teuthology.parallel:result is None 2020-03-28T06:17:05.726 INFO:teuthology.orchestra.run.smithi168.stdout:Resolving Dependencies 2020-03-28T06:17:05.726 INFO:teuthology.orchestra.run.smithi168.stdout:--> Running transaction check 2020-03-28T06:17:05.726 INFO:teuthology.orchestra.run.smithi168.stdout:---> Package rbd-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 will be installed 2020-03-28T06:17:06.108 INFO:teuthology.orchestra.run.smithi168.stdout:--> Finished Dependency Resolution 2020-03-28T06:17:06.339 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:06.339 INFO:teuthology.orchestra.run.smithi168.stdout:Dependencies Resolved 2020-03-28T06:17:06.340 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:06.340 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:17:06.341 INFO:teuthology.orchestra.run.smithi168.stdout: Package Arch Version Repository 2020-03-28T06:17:06.341 INFO:teuthology.orchestra.run.smithi168.stdout: Size 2020-03-28T06:17:06.341 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:17:06.341 INFO:teuthology.orchestra.run.smithi168.stdout:Installing: 2020-03-28T06:17:06.341 INFO:teuthology.orchestra.run.smithi168.stdout: rbd-fuse x86_64 2:14.2.8-334.gcd131a1eb0.el7 ceph 75 k 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction Summary 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout:================================================================================ 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout:Install 1 Package 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout:Total download size: 75 k 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout:Installed size: 196 k 2020-03-28T06:17:06.342 INFO:teuthology.orchestra.run.smithi168.stdout:Downloading packages: 2020-03-28T06:17:07.082 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction check 2020-03-28T06:17:07.106 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction test 2020-03-28T06:17:07.108 INFO:teuthology.orchestra.run.smithi168.stdout:Transaction test succeeded 2020-03-28T06:17:07.109 INFO:teuthology.orchestra.run.smithi168.stdout:Running transaction 2020-03-28T06:17:07.719 INFO:teuthology.orchestra.run.smithi168.stdout: Installing : 2:rbd-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:08.218 INFO:teuthology.orchestra.run.smithi168.stdout: Verifying : 2:rbd-fuse-14.2.8-334.gcd131a1eb0.el7.x86_64 1/1 2020-03-28T06:17:08.218 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:08.218 INFO:teuthology.orchestra.run.smithi168.stdout:Installed: 2020-03-28T06:17:08.218 INFO:teuthology.orchestra.run.smithi168.stdout: rbd-fuse.x86_64 2:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:08.218 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:17:08.219 INFO:teuthology.orchestra.run.smithi168.stdout:Complete! 2020-03-28T06:17:08.347 DEBUG:teuthology.parallel:result is None 2020-03-28T06:17:08.347 INFO:teuthology.orchestra.run.smithi156:> sudo lsb_release -is 2020-03-28T06:17:08.495 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS 2020-03-28T06:17:08.497 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-28T06:17:08.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=cd131a1eb0d13cbc9e432c4911cf5025d18aef1e 2020-03-28T06:17:08.839 INFO:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-28T06:17:08.964 INFO:teuthology.orchestra.run.smithi156.stdout:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:08.964 INFO:teuthology.packaging:The installed version of ceph is 14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:08.965 INFO:teuthology.task.install:The correct ceph version 14.2.8-334.gcd131a1eb0 is installed. 2020-03-28T06:17:08.965 INFO:teuthology.orchestra.run.smithi168:> sudo lsb_release -is 2020-03-28T06:17:09.116 INFO:teuthology.orchestra.run.smithi168.stdout:CentOS 2020-03-28T06:17:09.118 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-28T06:17:09.119 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=cd131a1eb0d13cbc9e432c4911cf5025d18aef1e 2020-03-28T06:17:09.361 INFO:teuthology.orchestra.run.smithi168:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-28T06:17:09.494 INFO:teuthology.orchestra.run.smithi168.stdout:14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:09.495 INFO:teuthology.packaging:The installed version of ceph is 14.2.8-334.gcd131a1eb0.el7 2020-03-28T06:17:09.495 INFO:teuthology.task.install:The correct ceph version 14.2.8-334.gcd131a1eb0 is installed. 2020-03-28T06:17:09.495 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-28T06:17:09.495 INFO:teuthology.orchestra.run.smithi156:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-28T06:17:09.616 INFO:teuthology.orchestra.run.smithi168:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-28T06:17:09.737 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-28T06:17:09.738 INFO:teuthology.orchestra.run.smithi156:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-28T06:17:09.856 INFO:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-28T06:17:09.969 INFO:teuthology.orchestra.run.smithi168:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-28T06:17:10.089 INFO:teuthology.orchestra.run.smithi168:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-28T06:17:10.195 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-28T06:17:10.196 INFO:teuthology.orchestra.run.smithi156:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-28T06:17:10.316 INFO:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-28T06:17:10.421 INFO:teuthology.orchestra.run.smithi168:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-28T06:17:10.544 INFO:teuthology.orchestra.run.smithi168:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-28T06:17:10.649 INFO:teuthology.run_tasks:Running task ceph... 2020-03-28T06:17:10.700 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-03-28T06:17:10.700 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2020-03-28T06:17:10.704 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 777 /var/log/ceph 2020-03-28T06:17:10.821 INFO:tasks.ceph:Disabling ceph logrotate... 2020-03-28T06:17:10.821 INFO:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2020-03-28T06:17:10.825 INFO:teuthology.orchestra.run.smithi168:> sudo rm -f -- /etc/logrotate.d/ceph 2020-03-28T06:17:10.933 INFO:tasks.ceph:Creating extra log directories... 2020-03-28T06:17:10.933 INFO:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-03-28T06:17:10.936 INFO:teuthology.orchestra.run.smithi168:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-03-28T06:17:11.029 INFO:tasks.ceph:Creating ceph cluster c1... 2020-03-28T06:17:11.029 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'c1', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'global': {'osd heartbeat grace': 40}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}, 'osd': {'debug osd': 25, 'osd fast shutdown': False, 'debug journal': 20, 'debug ms': 1, 'debug filestore': 20}, 'client': {'rgw crypt require ssl': False, 'rgw md log max shards': 4, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw data log num shards': 4, 'debug rgw': 20, 'rgw sync log trim interval': 0, 'rgw curl low speed time': 300}}} 2020-03-28T06:17:11.029 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'wait-for-scrub': False, 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}, 'cluster': 'c1', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'conf': {'global': {'osd heartbeat grace': 40}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}, 'osd': {'debug osd': 25, 'osd fast shutdown': False, 'debug journal': 20, 'debug ms': 1, 'debug filestore': 20}, 'client': {'rgw crypt require ssl': False, 'rgw md log max shards': 4, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw data log num shards': 4, 'debug rgw': 20, 'rgw sync log trim interval': 0, 'rgw curl low speed time': 300}}}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c2.client.1': {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c2.client.0': {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c1.client.0': {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c1.client.1': {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.2393', 'owner': 'scheduled_kchai@teuthology', 'targets': {'smithi156.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVeY4umzW6UfyAQyF0mX5UySUYTGZr8AD8EWx3EzUiYESaXmRAx3cQSAbHtKkDkRQbTxFtAeOPifOGCXC1jA9BZHFSd8MAs/jswb2FmBxT60nVo6DtjeqWqjb4+7jAeNxo1Xj5fh+bpJKUE+XxXCWCgmtqsMHaBXN1nixUZwHtRE39ZZjfKLsAu45i122hiC9Mv7FKIhNyMSf+QqRX6jypdhrnMe8zHN2EuuFmT4GU/S7gok/a/mCWKf9FBber1PEVgW9FVmLCD26odAm6EnnA9sdVUW3cPxCxi4cYAU7/Bp7pqlmxTTC5Yir2ROfxun64//YQbMvxw/Ta765joJ9b', 'smithi168.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKIS5Dl34X/BN+J6deeiBJbAans534cCmppZyE7VNs0qxn6GJKDt0ONQ39VC2Hu/4nXeZepysnZ2kXoNXxiERPx+Jg0CHqE8VpXRuGLpJaHfm1saOesRPQcjD1h77kMdoyPKrRG2wGtsSqtPss6hF22R5RzhCspU493aK7JRKCv/lclpkrjBrZcw+gp7Vpd1zRTj0cBB7y51kC9lYB1HjL/j9d1mulBDvqIDWwPTHw7lazcMXO7ATxHTN7v94rWJQvoJr3l6r14DqEwcIK+1b9yu+wXnE1tSpBx26S7AicxpHfVqtflDQQcgGi0/bW2/3ZKL7XsjwLaqTgKYHqvxhp'}, 'job_id': '4896252', 'tube': 'smithi', 'suite_branch': 'nautilus-kefu2-testing-2020-03-27-2234', 'priority': 90, 'first_in_suite': False, 'branch': 'nautilus-kefu2-testing-2020-03-27-2234', 'suite': 'rgw', 'email': 'tchaikov@gmail.com', 'archive_path': '/home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252', 'description': 'rgw/multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'name': 'kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'os_type': 'centos', 'overrides': {'ceph': {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'wait-for-scrub': False, 'conf': {'global': {'osd heartbeat grace': 40}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}, 'osd': {'debug osd': 25, 'osd fast shutdown': False, 'debug journal': 20, 'debug ms': 1, 'debug filestore': 20}, 'client': {'rgw crypt require ssl': False, 'rgw md log max shards': 4, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw data log num shards': 4, 'debug rgw': 20, 'rgw sync log trim interval': 0, 'rgw curl low speed time': 300}}, 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'notcmalloc', 'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'zonegroups': [{'is_default': True, 'is_master': True, 'zones': [{'is_default': True, 'is_master': True, 'endpoints': ['c1.client.0'], 'name': 'test-zone1'}, {'is_default': True, 'endpoints': ['c2.client.0'], 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}], 'endpoints': ['c1.client.0'], 'name': 'test-zonegroup'}], 'realm': {'is default': True, 'name': 'test-realm'}}, 'admin_socket': {'branch': 'nautilus-kefu2-testing-2020-03-27-2234'}, 'workunit': {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'branch': 'nautilus-kefu2-testing-2020-03-27-2234'}}} 2020-03-28T06:17:11.031 INFO:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-03-28T06:17:11.035 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-03-28T06:17:11.107 INFO:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2020-03-28T06:17:11.110 INFO:teuthology.orchestra.run.smithi168:> sudo install -d -m0777 -- /var/run/ceph 2020-03-28T06:17:11.298 DEBUG:teuthology.orchestra.remote:smithi156:/scratch_devs is 72B 2020-03-28T06:17:11.392 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-28T06:17:11.392 INFO:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-03-28T06:17:11.478 INFO:teuthology.orchestra.run.smithi156.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2020-03-28T06:17:11.478 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:17:11.478 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 108182 Links: 1 2020-03-28T06:17:11.479 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:17:11.479 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:17:11.479 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2020-03-28 06:16:07.609323676 +0000 2020-03-28T06:17:11.479 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2020-03-28 06:14:23.172784582 +0000 2020-03-28T06:17:11.480 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2020-03-28 06:14:23.172784582 +0000 2020-03-28T06:17:11.480 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2020-03-28T06:17:11.505 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2020-03-28T06:17:11.505 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2020-03-28T06:17:11.506 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes (512 B) copied, 0.000471647 s, 1.1 MB/s 2020-03-28T06:17:11.511 INFO:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-03-28T06:17:11.575 INFO:teuthology.orchestra.run.smithi156.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2020-03-28T06:17:11.575 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:17:11.575 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 107339 Links: 1 2020-03-28T06:17:11.576 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:17:11.576 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:17:11.576 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2020-03-28 06:16:07.609323676 +0000 2020-03-28T06:17:11.576 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2020-03-28 06:14:23.661776713 +0000 2020-03-28T06:17:11.576 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2020-03-28 06:14:23.661776713 +0000 2020-03-28T06:17:11.577 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2020-03-28T06:17:11.596 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2020-03-28T06:17:11.597 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2020-03-28T06:17:11.597 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes (512 B) copied, 0.00041635 s, 1.2 MB/s 2020-03-28T06:17:11.601 INFO:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-03-28T06:17:11.661 INFO:teuthology.orchestra.run.smithi156.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2020-03-28T06:17:11.661 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:17:11.661 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 106073 Links: 1 2020-03-28T06:17:11.661 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:17:11.661 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:17:11.661 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2020-03-28 06:16:07.609323676 +0000 2020-03-28T06:17:11.661 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2020-03-28 06:14:24.107769536 +0000 2020-03-28T06:17:11.662 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2020-03-28 06:14:24.107769536 +0000 2020-03-28T06:17:11.662 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2020-03-28T06:17:11.679 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2020-03-28T06:17:11.679 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2020-03-28T06:17:11.680 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes (512 B) copied, 0.000304348 s, 1.7 MB/s 2020-03-28T06:17:11.684 INFO:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-03-28T06:17:11.735 INFO:teuthology.orchestra.run.smithi156.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2020-03-28T06:17:11.735 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:17:11.735 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 100052 Links: 1 2020-03-28T06:17:11.735 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:17:11.735 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:17:11.735 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2020-03-28 06:16:07.609323676 +0000 2020-03-28T06:17:11.736 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2020-03-28 06:14:24.594761700 +0000 2020-03-28T06:17:11.736 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2020-03-28 06:14:24.594761700 +0000 2020-03-28T06:17:11.736 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2020-03-28T06:17:11.752 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2020-03-28T06:17:11.752 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2020-03-28T06:17:11.752 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes (512 B) copied, 0.000357363 s, 1.4 MB/s 2020-03-28T06:17:11.756 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-03-28T06:17:11.756 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-28T06:17:11.756 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-03-28T06:17:11.757 INFO:tasks.ceph:dev map: {'c1.osd.2': '/dev/vg_nvme/lv_2', 'c1.osd.0': '/dev/vg_nvme/lv_4', 'c1.osd.1': '/dev/vg_nvme/lv_1'} 2020-03-28T06:17:11.757 INFO:tasks.ceph:Generating config... 2020-03-28T06:17:11.758 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-03-28T06:17:11.758 INFO:tasks.ceph:[mon] debug mon = 20 2020-03-28T06:17:11.758 INFO:tasks.ceph:[mon] debug paxos = 20 2020-03-28T06:17:11.758 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-03-28T06:17:11.758 INFO:tasks.ceph:[mon] debug ms = 1 2020-03-28T06:17:11.758 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-03-28T06:17:11.758 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-03-28T06:17:11.758 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-03-28T06:17:11.759 INFO:tasks.ceph:[osd] debug osd = 25 2020-03-28T06:17:11.759 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-03-28T06:17:11.759 INFO:tasks.ceph:[osd] debug journal = 20 2020-03-28T06:17:11.759 INFO:tasks.ceph:[osd] debug ms = 1 2020-03-28T06:17:11.759 INFO:tasks.ceph:[osd] debug filestore = 20 2020-03-28T06:17:11.759 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-03-28T06:17:11.759 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-03-28T06:17:11.759 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-03-28T06:17:11.760 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-03-28T06:17:11.760 INFO:tasks.ceph:[client] debug rgw = 20 2020-03-28T06:17:11.760 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-03-28T06:17:11.760 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-03-28T06:17:11.760 INFO:tasks.ceph:Setting up c1.mon.a... 2020-03-28T06:17:11.760 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-03-28T06:17:11.912 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/c1.keyring 2020-03-28T06:17:11.916 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-03-28T06:17:12.091 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-03-28T06:17:12.188 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.156')] 2020-03-28T06:17:12.188 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.156 --print /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:12.325 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:12.325 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: generated fsid ff1cba1e-a6a7-4868-8184-daa87358bc60 2020-03-28T06:17:12.325 INFO:teuthology.orchestra.run.smithi156.stdout:epoch 0 2020-03-28T06:17:12.325 INFO:teuthology.orchestra.run.smithi156.stdout:fsid ff1cba1e-a6a7-4868-8184-daa87358bc60 2020-03-28T06:17:12.326 INFO:teuthology.orchestra.run.smithi156.stdout:last_changed 2020-03-28 06:17:12.323554 2020-03-28T06:17:12.326 INFO:teuthology.orchestra.run.smithi156.stdout:created 2020-03-28 06:17:12.323554 2020-03-28T06:17:12.326 INFO:teuthology.orchestra.run.smithi156.stdout:min_mon_release 0 (unknown) 2020-03-28T06:17:12.326 INFO:teuthology.orchestra.run.smithi156.stdout:0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2020-03-28T06:17:12.326 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-03-28T06:17:12.327 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID ff1cba1e-a6a7-4868-8184-daa87358bc60... 2020-03-28T06:17:12.328 INFO:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf 2020-03-28T06:17:12.332 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf 2020-03-28T06:17:12.543 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-03-28T06:17:12.543 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-03-28T06:17:12.673 INFO:tasks.ceph:Copying monmap to all nodes... 2020-03-28T06:17:12.737 DEBUG:teuthology.orchestra.remote:smithi156:/etc/ceph/c1.keyring is 206B 2020-03-28T06:17:12.853 DEBUG:teuthology.orchestra.remote:smithi156:/home/ubuntu/cephtest/c1.monmap is 177B 2020-03-28T06:17:12.911 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:17:12.911 INFO:teuthology.orchestra.run.smithi156:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-03-28T06:17:13.001 INFO:teuthology.orchestra.run.smithi156:> cat > /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:13.049 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:17:13.050 INFO:teuthology.orchestra.run.smithi168:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-03-28T06:17:13.198 INFO:teuthology.orchestra.run.smithi168:> cat > /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:13.262 INFO:tasks.ceph:Setting up mon nodes... 2020-03-28T06:17:13.262 INFO:tasks.ceph:Setting up mgr nodes... 2020-03-28T06:17:13.262 INFO:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-03-28T06:17:13.468 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-03-28T06:17:13.473 INFO:tasks.ceph:Setting up mds nodes... 2020-03-28T06:17:13.474 INFO:tasks.ceph_client:Setting up client nodes... 2020-03-28T06:17:13.474 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-03-28T06:17:13.655 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/c1.client.0.keyring 2020-03-28T06:17:13.693 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-03-28T06:17:13.853 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/c1.client.1.keyring 2020-03-28T06:17:13.885 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-03-28T06:17:13.885 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'c1.osd.2': '/dev/vg_nvme/lv_2', 'c1.osd.0': '/dev/vg_nvme/lv_4', 'c1.osd.1': '/dev/vg_nvme/lv_1'}} 2020-03-28T06:17:13.886 INFO:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-03-28T06:17:13.976 INFO:tasks.ceph:{'c1.osd.2': '/dev/vg_nvme/lv_2', 'c1.osd.0': '/dev/vg_nvme/lv_4', 'c1.osd.1': '/dev/vg_nvme/lv_1'} 2020-03-28T06:17:13.976 INFO:tasks.ceph:{} 2020-03-28T06:17:13.976 INFO:tasks.ceph:c1.osd.0 2020-03-28T06:17:13.977 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:17:13.977 INFO:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-03-28T06:17:14.545 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-03-28T06:17:14.545 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-28T06:17:14.545 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=0, sparse=0 2020-03-28T06:17:14.546 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-28T06:17:14.546 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2020-03-28T06:17:14.546 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-03-28T06:17:14.546 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-28T06:17:14.546 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-28T06:17:14.546 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-28T06:17:14.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2020-03-28T06:17:14.548 INFO:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/c1-0 2020-03-28T06:17:14.724 INFO:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-03-28T06:17:14.814 INFO:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-03-28T06:17:14.885 INFO:tasks.ceph:{'c1.osd.2': '/dev/vg_nvme/lv_2', 'c1.osd.0': '/dev/vg_nvme/lv_4', 'c1.osd.1': '/dev/vg_nvme/lv_1'} 2020-03-28T06:17:14.885 INFO:tasks.ceph:{} 2020-03-28T06:17:14.886 INFO:tasks.ceph:c1.osd.1 2020-03-28T06:17:14.886 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:17:14.886 INFO:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-03-28T06:17:15.426 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-03-28T06:17:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-28T06:17:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=0, sparse=0 2020-03-28T06:17:15.427 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-28T06:17:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2020-03-28T06:17:15.427 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-03-28T06:17:15.427 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-28T06:17:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-28T06:17:15.427 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-28T06:17:15.428 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2020-03-28T06:17:15.428 INFO:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-1 2020-03-28T06:17:15.562 INFO:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-03-28T06:17:15.653 INFO:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-03-28T06:17:15.727 INFO:tasks.ceph:{'c1.osd.2': '/dev/vg_nvme/lv_2', 'c1.osd.0': '/dev/vg_nvme/lv_4', 'c1.osd.1': '/dev/vg_nvme/lv_1'} 2020-03-28T06:17:15.727 INFO:tasks.ceph:{} 2020-03-28T06:17:15.727 INFO:tasks.ceph:c1.osd.2 2020-03-28T06:17:15.727 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:17:15.727 INFO:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-03-28T06:17:16.303 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-03-28T06:17:16.303 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-28T06:17:16.304 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=0, sparse=0 2020-03-28T06:17:16.304 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-28T06:17:16.304 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2020-03-28T06:17:16.304 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-03-28T06:17:16.304 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-28T06:17:16.304 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-28T06:17:16.305 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-28T06:17:16.306 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2020-03-28T06:17:16.306 INFO:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-2 2020-03-28T06:17:16.443 INFO:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-03-28T06:17:16.545 INFO:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:16.734 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:16.731 7f10348009c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-03-28T06:17:16.739 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:16.736 7f10348009c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-03-28T06:17:16.739 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:16.737 7f10348009c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-03-28T06:17:16.739 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:16.737 7f10348009c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-03-28T06:17:16.739 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:16.737 7f10348009c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-03-28T06:17:16.747 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:16.745 7f10348009c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-03-28T06:17:18.794 INFO:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-03-28T06:17:18.919 INFO:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:19.091 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:19.089 7f52867749c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-03-28T06:17:19.092 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:19.089 7f52867749c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-03-28T06:17:19.092 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:19.089 7f52867749c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-03-28T06:17:19.092 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:19.089 7f52867749c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-03-28T06:17:19.092 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:19.089 7f52867749c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-03-28T06:17:19.093 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:19.090 7f52867749c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-03-28T06:17:21.148 INFO:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-03-28T06:17:21.259 INFO:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:21.422 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:21.420 7f83e70849c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-03-28T06:17:21.423 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:21.420 7f83e70849c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-03-28T06:17:21.423 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:21.420 7f83e70849c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-03-28T06:17:21.423 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:21.420 7f83e70849c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-03-28T06:17:21.423 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:21.420 7f83e70849c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-03-28T06:17:21.424 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:17:21.421 7f83e70849c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-03-28T06:17:23.471 INFO:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-03-28T06:17:23.586 INFO:tasks.ceph:Reading keys from all nodes... 2020-03-28T06:17:23.587 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:17:23.655 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.p7CrtY2oDp 2020-03-28T06:17:23.656 INFO:teuthology.orchestra.run.smithi156:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.p7CrtY2oDp 2020-03-28T06:17:23.739 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.p7CrtY2oDp 2020-03-28T06:17:23.867 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.p7CrtY2oDp is 56B 2020-03-28T06:17:23.923 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.p7CrtY2oDp 2020-03-28T06:17:23.974 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:17:24.072 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.98OWv8EBf4 2020-03-28T06:17:24.072 INFO:teuthology.orchestra.run.smithi156:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.98OWv8EBf4 2020-03-28T06:17:24.177 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.98OWv8EBf4 2020-03-28T06:17:24.329 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.98OWv8EBf4 is 56B 2020-03-28T06:17:24.393 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.98OWv8EBf4 2020-03-28T06:17:24.449 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:17:24.571 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.sXC8NINMmn 2020-03-28T06:17:24.572 INFO:teuthology.orchestra.run.smithi156:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.sXC8NINMmn 2020-03-28T06:17:24.680 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.sXC8NINMmn 2020-03-28T06:17:24.860 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.sXC8NINMmn is 56B 2020-03-28T06:17:24.924 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.sXC8NINMmn 2020-03-28T06:17:24.994 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:17:25.124 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.Yz9EZVzNSi 2020-03-28T06:17:25.125 INFO:teuthology.orchestra.run.smithi156:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.Yz9EZVzNSi 2020-03-28T06:17:25.229 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.Yz9EZVzNSi 2020-03-28T06:17:25.413 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.Yz9EZVzNSi is 56B 2020-03-28T06:17:25.487 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.Yz9EZVzNSi 2020-03-28T06:17:25.654 DEBUG:teuthology.orchestra.remote:smithi156:/etc/ceph/c1.client.0.keyring is 59B 2020-03-28T06:17:25.779 DEBUG:teuthology.orchestra.remote:smithi156:/etc/ceph/c1.client.1.keyring is 59B 2020-03-28T06:17:25.854 INFO:tasks.ceph:Adding keys to all mons... 2020-03-28T06:17:25.854 INFO:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/c1.keyring 2020-03-28T06:17:25.945 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2020-03-28T06:17:25.945 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDp635euYrbGxAAvJAOahKzWLgIIMJUG6ecqQ== 2020-03-28T06:17:25.945 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2020-03-28T06:17:25.945 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDs635eFZetKxAAr0OAzGfP/9xwvRCFC8TeDQ== 2020-03-28T06:17:25.946 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2020-03-28T06:17:25.946 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDv635e19pdBRAAxUNZc/tUQdhbvMaGx4w9Dg== 2020-03-28T06:17:25.946 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2020-03-28T06:17:25.946 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDx635e5RMWGRAAGOulxO4AA81471GUMMHa+g== 2020-03-28T06:17:25.946 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2020-03-28T06:17:25.946 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDp635e8cABJxAA/yXfh5CSmSHQ9gqGZ2Uc+Q== 2020-03-28T06:17:25.947 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2020-03-28T06:17:25.947 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQDp635eDITOMhAAeJju+RLBHihBQALVBLrrkw== 2020-03-28T06:17:25.948 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-28T06:17:26.123 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-28T06:17:26.304 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-28T06:17:26.506 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-28T06:17:26.686 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-03-28T06:17:26.887 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-03-28T06:17:27.049 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-03-28T06:17:27.049 INFO:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-03-28T06:17:27.140 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-03-28T06:17:27.348 INFO:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-03-28T06:17:27.449 INFO:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-03-28T06:17:27.522 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-03-28T06:17:27.522 DEBUG:teuthology.misc:running c1.mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c1.mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2020-03-28T06:17:27.523 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-03-28T06:17:27.523 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster c1 -i a 2020-03-28T06:17:27.526 INFO:tasks.ceph.c1.mon.a:Started 2020-03-28T06:17:27.527 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-03-28T06:17:27.527 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-03-28T06:17:27.527 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster c1 -i x 2020-03-28T06:17:27.530 INFO:tasks.ceph.c1.mgr.x:Started 2020-03-28T06:17:27.530 INFO:tasks.ceph:Setting crush tunables to default 2020-03-28T06:17:27.530 INFO:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c1 osd crush tunables default 2020-03-28T06:17:33.140 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:00:05.111 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:17:39.285 INFO:teuthology.orchestra.run.smithi156.stderr:adjusted tunables profile to default 2020-03-28T06:17:39.303 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-03-28T06:17:39.303 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:17:39.360 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.sYkMyzbvZD 2020-03-28T06:17:39.360 INFO:teuthology.orchestra.run.smithi156:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.sYkMyzbvZD 2020-03-28T06:17:39.441 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.sYkMyzbvZD 2020-03-28T06:17:39.546 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.sYkMyzbvZD is 37B 2020-03-28T06:17:39.596 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.sYkMyzbvZD 2020-03-28T06:17:39.639 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:17:39.728 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.klE0Gpfl8B 2020-03-28T06:17:39.729 INFO:teuthology.orchestra.run.smithi156:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.klE0Gpfl8B 2020-03-28T06:17:39.794 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.klE0Gpfl8B 2020-03-28T06:17:39.893 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.klE0Gpfl8B is 37B 2020-03-28T06:17:39.939 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.klE0Gpfl8B 2020-03-28T06:17:39.981 INFO:teuthology.orchestra.run.smithi156:> mktemp 2020-03-28T06:17:40.062 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.FTIIMP0bDd 2020-03-28T06:17:40.062 INFO:teuthology.orchestra.run.smithi156:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.FTIIMP0bDd 2020-03-28T06:17:40.173 INFO:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.FTIIMP0bDd 2020-03-28T06:17:40.283 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.FTIIMP0bDd is 37B 2020-03-28T06:17:40.329 INFO:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.FTIIMP0bDd 2020-03-28T06:17:40.369 INFO:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c1 osd new 7e6c4a16-87d9-4f7a-b8b9-39cdf058358b 0 2020-03-28T06:17:41.245 INFO:teuthology.orchestra.run.smithi156.stdout:0 2020-03-28T06:17:41.268 INFO:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c1 osd new 4e839d62-7a39-4695-a0e7-b9e5fdad7809 1 2020-03-28T06:17:42.447 INFO:teuthology.orchestra.run.smithi156.stdout:1 2020-03-28T06:17:42.472 INFO:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c1 osd new 6963fa03-1a03-40cb-9de7-4db60651d94e 2 2020-03-28T06:17:43.534 INFO:teuthology.orchestra.run.smithi156.stdout:2 2020-03-28T06:17:43.552 DEBUG:teuthology.misc:running c1.osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c1.osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2020-03-28T06:17:43.552 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-03-28T06:17:43.553 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.osd.0.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c1 -i 0 2020-03-28T06:17:43.556 INFO:tasks.ceph.c1.osd.0:Started 2020-03-28T06:17:43.557 DEBUG:teuthology.misc:running c1.osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c1.osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2020-03-28T06:17:43.557 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-03-28T06:17:43.557 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.osd.1.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c1 -i 1 2020-03-28T06:17:43.560 INFO:tasks.ceph.c1.osd.1:Started 2020-03-28T06:17:43.560 DEBUG:teuthology.misc:running c1.osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c1.osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2020-03-28T06:17:43.561 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-03-28T06:17:43.561 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.osd.2.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c1 -i 2 2020-03-28T06:17:43.563 INFO:tasks.ceph.c1.osd.2:Started 2020-03-28T06:17:43.564 INFO:tasks.ceph:Waiting for OSDs to come up 2020-03-28T06:17:43.867 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-03-28T06:17:44.665 INFO:teuthology.misc.health.smithi156.stdout: 2020-03-28T06:17:44.665 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":5,"fsid":"ff1cba1e-a6a7-4868-8184-daa87358bc60","created":"2020-03-28 06:17:35.813323","modified":"2020-03-28 06:17:43.432203","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"7e6c4a16-87d9-4f7a-b8b9-39cdf058358b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"4e839d62-7a39-4695-a0e7-b9e5fdad7809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"6963fa03-1a03-40cb-9de7-4db60651d94e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:17:44.678 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-03-28T06:17:49.759 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:==00:00:00:06.066 8986== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:49.814 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:==00:00:00:06.121 8986== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:50.434 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:2020-03-28 06:17:50.416 9ae0e40 -1 Falling back to public interface 2020-03-28T06:17:50.630 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:==00:00:00:06.937 8986== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:50.747 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:==00:00:00:07.057 8985== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:50.798 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:==00:00:00:07.108 8984== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:50.801 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:==00:00:00:07.112 8985== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:50.859 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:==00:00:00:07.169 8984== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:50.981 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-03-28T06:17:51.046 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:==00:00:00:07.354 8986== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:51.057 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:==00:00:00:07.365 8986== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:51.374 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:2020-03-28 06:17:51.361 9ae0e40 -1 Falling back to public interface 2020-03-28T06:17:51.424 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:2020-03-28 06:17:51.411 9ae0e40 -1 Falling back to public interface 2020-03-28T06:17:51.617 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:==00:00:00:07.927 8984== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:51.659 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:==00:00:00:07.969 8985== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:51.785 INFO:teuthology.misc.health.smithi156.stdout: 2020-03-28T06:17:51.785 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":5,"fsid":"ff1cba1e-a6a7-4868-8184-daa87358bc60","created":"2020-03-28 06:17:35.813323","modified":"2020-03-28 06:17:43.432203","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"7e6c4a16-87d9-4f7a-b8b9-39cdf058358b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"4e839d62-7a39-4695-a0e7-b9e5fdad7809","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"6963fa03-1a03-40cb-9de7-4db60651d94e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:17:51.799 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-03-28T06:17:52.033 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:==00:00:00:08.343 8985== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:52.042 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:==00:00:00:08.353 8984== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:52.050 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:==00:00:00:08.360 8985== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:52.059 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:==00:00:00:08.369 8984== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:52.144 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:==00:00:00:08.452 8986== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:53.092 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:==00:00:00:09.403 8985== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:53.262 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:==00:00:00:09.572 8984== Warning: unimplemented fcntl command: 1038 2020-03-28T06:17:53.634 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:2020-03-28 06:17:53.629 9ae0e40 -1 osd.2 0 log_to_monitors {default=true} 2020-03-28T06:17:54.314 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:2020-03-28 06:17:54.309 9ae0e40 -1 osd.1 0 log_to_monitors {default=true} 2020-03-28T06:17:54.679 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:2020-03-28 06:17:54.673 9ae0e40 -1 osd.0 0 log_to_monitors {default=true} 2020-03-28T06:17:55.488 INFO:tasks.ceph.c1.osd.2.smithi156.stderr:2020-03-28 06:17:55.475 288b7700 -1 osd.2 0 waiting for initial osdmap 2020-03-28T06:17:56.525 INFO:tasks.ceph.c1.osd.0.smithi156.stderr:2020-03-28 06:17:56.515 288b7700 -1 osd.0 0 waiting for initial osdmap 2020-03-28T06:17:56.568 INFO:tasks.ceph.c1.osd.1.smithi156.stderr:2020-03-28 06:17:56.551 288b7700 -1 osd.1 0 waiting for initial osdmap 2020-03-28T06:17:58.103 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-03-28T06:17:59.068 INFO:teuthology.misc.health.smithi156.stdout: 2020-03-28T06:17:59.068 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":9,"fsid":"ff1cba1e-a6a7-4868-8184-daa87358bc60","created":"2020-03-28 06:17:35.813323","modified":"2020-03-28 06:17:57.298801","last_up_change":"2020-03-28 06:17:57.298801","last_in_change":"2020-03-28 06:17:57.298801","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"7e6c4a16-87d9-4f7a-b8b9-39cdf058358b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":8984},{"type":"v1","addr":"172.21.15.156:6819","nonce":8984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":8984},{"type":"v1","addr":"172.21.15.156:6821","nonce":8984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":8984},{"type":"v1","addr":"172.21.15.156:6825","nonce":8984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":8984},{"type":"v1","addr":"172.21.15.156:6823","nonce":8984}]},"public_addr":"172.21.15.156:6819/8984","cluster_addr":"172.21.15.156:6821/8984","heartbeat_back_addr":"172.21.15.156:6825/8984","heartbeat_front_addr":"172.21.15.156:6823/8984","state":["exists","up"]},{"osd":1,"uuid":"4e839d62-7a39-4695-a0e7-b9e5fdad7809","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":8985},{"type":"v1","addr":"172.21.15.156:6811","nonce":8985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":8985},{"type":"v1","addr":"172.21.15.156:6813","nonce":8985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":8985},{"type":"v1","addr":"172.21.15.156:6817","nonce":8985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":8985},{"type":"v1","addr":"172.21.15.156:6815","nonce":8985}]},"public_addr":"172.21.15.156:6811/8985","cluster_addr":"172.21.15.156:6813/8985","heartbeat_back_addr":"172.21.15.156:6817/8985","heartbeat_front_addr":"172.21.15.156:6815/8985","state":["exists","up"]},{"osd":2,"uuid":"6963fa03-1a03-40cb-9de7-4db60651d94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":8986},{"type":"v1","addr":"172.21.15.156:6803","nonce":8986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":8986},{"type":"v1","addr":"172.21.15.156:6805","nonce":8986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":8986},{"type":"v1","addr":"172.21.15.156:6809","nonce":8986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":8986},{"type":"v1","addr":"172.21.15.156:6807","nonce":8986}]},"public_addr":"172.21.15.156:6803/8986","cluster_addr":"172.21.15.156:6805/8986","heartbeat_back_addr":"172.21.15.156:6809/8986","heartbeat_front_addr":"172.21.15.156:6807/8986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:17:59.090 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-03-28T06:17:59.090 INFO:tasks.ceph:Creating RBD pool 2020-03-28T06:17:59.090 INFO:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c1 osd pool create rbd 8 2020-03-28T06:18:00.838 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'rbd' created 2020-03-28T06:18:00.860 INFO:teuthology.orchestra.run.smithi156:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-03-28T06:18:02.777 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rbd' on pool 'rbd' 2020-03-28T06:18:02.795 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-03-28T06:18:02.796 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 osd dump --format=json 2020-03-28T06:18:03.499 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:18:03.500 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":12,"fsid":"ff1cba1e-a6a7-4868-8184-daa87358bc60","created":"2020-03-28 06:17:35.813323","modified":"2020-03-28 06:18:02.635253","last_up_change":"2020-03-28 06:17:57.298801","last_in_change":"2020-03-28 06:17:57.298801","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-03-28 06:17:59.994443","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7e6c4a16-87d9-4f7a-b8b9-39cdf058358b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":8984},{"type":"v1","addr":"172.21.15.156:6819","nonce":8984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":8984},{"type":"v1","addr":"172.21.15.156:6821","nonce":8984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":8984},{"type":"v1","addr":"172.21.15.156:6825","nonce":8984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":8984},{"type":"v1","addr":"172.21.15.156:6823","nonce":8984}]},"public_addr":"172.21.15.156:6819/8984","cluster_addr":"172.21.15.156:6821/8984","heartbeat_back_addr":"172.21.15.156:6825/8984","heartbeat_front_addr":"172.21.15.156:6823/8984","state":["exists","up"]},{"osd":1,"uuid":"4e839d62-7a39-4695-a0e7-b9e5fdad7809","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":8985},{"type":"v1","addr":"172.21.15.156:6811","nonce":8985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":8985},{"type":"v1","addr":"172.21.15.156:6813","nonce":8985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":8985},{"type":"v1","addr":"172.21.15.156:6817","nonce":8985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":8985},{"type":"v1","addr":"172.21.15.156:6815","nonce":8985}]},"public_addr":"172.21.15.156:6811/8985","cluster_addr":"172.21.15.156:6813/8985","heartbeat_back_addr":"172.21.15.156:6817/8985","heartbeat_front_addr":"172.21.15.156:6815/8985","state":["exists","up"]},{"osd":2,"uuid":"6963fa03-1a03-40cb-9de7-4db60651d94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":8986},{"type":"v1","addr":"172.21.15.156:6803","nonce":8986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":8986},{"type":"v1","addr":"172.21.15.156:6805","nonce":8986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":8986},{"type":"v1","addr":"172.21.15.156:6809","nonce":8986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":8986},{"type":"v1","addr":"172.21.15.156:6807","nonce":8986}]},"public_addr":"172.21.15.156:6803/8986","cluster_addr":"172.21.15.156:6805/8986","heartbeat_back_addr":"172.21.15.156:6809/8986","heartbeat_front_addr":"172.21.15.156:6807/8986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:18:03.519 INFO:tasks.ceph.ceph_manager.c1:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-28 06:17:59.994443', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'warn', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'12', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-03-28T06:18:03.520 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 osd pool get rbd pg_num 2020-03-28T06:18:04.521 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 8 2020-03-28T06:18:04.547 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-03-28T06:18:04.547 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-03-28T06:18:04.547 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 mgr dump --format=json 2020-03-28T06:18:05.593 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:18:05.616 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":8059},{"type":"v1","addr":"172.21.15.156:6801","nonce":8059}]},"active_addr":"172.21.15.156:6801/8059","active_change":"2020-03-28 06:17:40.245487","available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"]}} 2020-03-28T06:18:05.618 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-03-28T06:18:05.921 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-03-28T06:18:06.910 INFO:teuthology.misc.health.smithi156.stdout: 2020-03-28T06:18:06.910 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":13,"fsid":"ff1cba1e-a6a7-4868-8184-daa87358bc60","created":"2020-03-28 06:17:35.813323","modified":"2020-03-28 06:18:03.819201","last_up_change":"2020-03-28 06:17:57.298801","last_in_change":"2020-03-28 06:17:57.298801","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-03-28 06:17:59.994443","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7e6c4a16-87d9-4f7a-b8b9-39cdf058358b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":8984},{"type":"v1","addr":"172.21.15.156:6819","nonce":8984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":8984},{"type":"v1","addr":"172.21.15.156:6821","nonce":8984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":8984},{"type":"v1","addr":"172.21.15.156:6825","nonce":8984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":8984},{"type":"v1","addr":"172.21.15.156:6823","nonce":8984}]},"public_addr":"172.21.15.156:6819/8984","cluster_addr":"172.21.15.156:6821/8984","heartbeat_back_addr":"172.21.15.156:6825/8984","heartbeat_front_addr":"172.21.15.156:6823/8984","state":["exists","up"]},{"osd":1,"uuid":"4e839d62-7a39-4695-a0e7-b9e5fdad7809","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":8985},{"type":"v1","addr":"172.21.15.156:6811","nonce":8985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":8985},{"type":"v1","addr":"172.21.15.156:6813","nonce":8985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":8985},{"type":"v1","addr":"172.21.15.156:6817","nonce":8985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":8985},{"type":"v1","addr":"172.21.15.156:6815","nonce":8985}]},"public_addr":"172.21.15.156:6811/8985","cluster_addr":"172.21.15.156:6813/8985","heartbeat_back_addr":"172.21.15.156:6817/8985","heartbeat_front_addr":"172.21.15.156:6815/8985","state":["exists","up"]},{"osd":2,"uuid":"6963fa03-1a03-40cb-9de7-4db60651d94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":8986},{"type":"v1","addr":"172.21.15.156:6803","nonce":8986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":8986},{"type":"v1","addr":"172.21.15.156:6805","nonce":8986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":8986},{"type":"v1","addr":"172.21.15.156:6809","nonce":8986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":8986},{"type":"v1","addr":"172.21.15.156:6807","nonce":8986}]},"public_addr":"172.21.15.156:6803/8986","cluster_addr":"172.21.15.156:6805/8986","heartbeat_back_addr":"172.21.15.156:6809/8986","heartbeat_front_addr":"172.21.15.156:6807/8986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:18:06.928 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-03-28T06:18:06.928 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 osd dump --format=json 2020-03-28T06:18:07.933 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:18:07.933 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":13,"fsid":"ff1cba1e-a6a7-4868-8184-daa87358bc60","created":"2020-03-28 06:17:35.813323","modified":"2020-03-28 06:18:03.819201","last_up_change":"2020-03-28 06:17:57.298801","last_in_change":"2020-03-28 06:17:57.298801","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-03-28 06:17:59.994443","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7e6c4a16-87d9-4f7a-b8b9-39cdf058358b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":8984},{"type":"v1","addr":"172.21.15.156:6819","nonce":8984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":8984},{"type":"v1","addr":"172.21.15.156:6821","nonce":8984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":8984},{"type":"v1","addr":"172.21.15.156:6825","nonce":8984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":8984},{"type":"v1","addr":"172.21.15.156:6823","nonce":8984}]},"public_addr":"172.21.15.156:6819/8984","cluster_addr":"172.21.15.156:6821/8984","heartbeat_back_addr":"172.21.15.156:6825/8984","heartbeat_front_addr":"172.21.15.156:6823/8984","state":["exists","up"]},{"osd":1,"uuid":"4e839d62-7a39-4695-a0e7-b9e5fdad7809","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":8985},{"type":"v1","addr":"172.21.15.156:6811","nonce":8985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":8985},{"type":"v1","addr":"172.21.15.156:6813","nonce":8985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":8985},{"type":"v1","addr":"172.21.15.156:6817","nonce":8985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":8985},{"type":"v1","addr":"172.21.15.156:6815","nonce":8985}]},"public_addr":"172.21.15.156:6811/8985","cluster_addr":"172.21.15.156:6813/8985","heartbeat_back_addr":"172.21.15.156:6817/8985","heartbeat_front_addr":"172.21.15.156:6815/8985","state":["exists","up"]},{"osd":2,"uuid":"6963fa03-1a03-40cb-9de7-4db60651d94e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":8986},{"type":"v1","addr":"172.21.15.156:6803","nonce":8986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":8986},{"type":"v1","addr":"172.21.15.156:6805","nonce":8986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":8986},{"type":"v1","addr":"172.21.15.156:6809","nonce":8986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":8986},{"type":"v1","addr":"172.21.15.156:6807","nonce":8986}]},"public_addr":"172.21.15.156:6803/8986","cluster_addr":"172.21.15.156:6805/8986","heartbeat_back_addr":"172.21.15.156:6809/8986","heartbeat_front_addr":"172.21.15.156:6807/8986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:18:07.957 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 tell osd.0 flush_pg_stats 2020-03-28T06:18:08.481 INFO:teuthology.orchestra.run.smithi156.stdout:38654705669 2020-03-28T06:18:08.495 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 tell osd.1 flush_pg_stats 2020-03-28T06:18:09.101 INFO:teuthology.orchestra.run.smithi156.stdout:38654705668 2020-03-28T06:18:09.103 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 tell osd.2 flush_pg_stats 2020-03-28T06:18:09.665 INFO:teuthology.orchestra.run.smithi156.stdout:34359738374 2020-03-28T06:18:09.687 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 osd last-stat-seq osd.0 2020-03-28T06:18:10.683 INFO:teuthology.orchestra.run.smithi156.stdout:38654705669 2020-03-28T06:18:10.707 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705669 got 38654705669 for osd.0 2020-03-28T06:18:10.707 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 osd last-stat-seq osd.1 2020-03-28T06:18:11.546 INFO:teuthology.orchestra.run.smithi156.stdout:38654705668 2020-03-28T06:18:11.561 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.1 2020-03-28T06:18:11.561 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 osd last-stat-seq osd.2 2020-03-28T06:18:12.444 INFO:teuthology.orchestra.run.smithi156.stdout:34359738374 2020-03-28T06:18:12.462 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738374 for osd.2 2020-03-28T06:18:12.462 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-03-28T06:18:12.463 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 pg dump --format=json 2020-03-28T06:18:13.274 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2020-03-28T06:18:13.274 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:18:13.293 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2020-03-28 06:18:12.339016","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":86703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.159513","last_change":"2020-03-28 06:18:03.079635","last_active":"2020-03-28 06:18:03.159513","last_peered":"2020-03-28 06:18:03.159513","last_clean":"2020-03-28 06:18:03.159513","last_became_active":"2020-03-28 06:18:03.063900","last_became_peered":"2020-03-28 06:18:03.063900","last_unstale":"2020-03-28 06:18:03.159513","last_undegraded":"2020-03-28 06:18:03.159513","last_fullsized":"2020-03-28 06:18:03.159513","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.160616","last_change":"2020-03-28 06:18:03.146905","last_active":"2020-03-28 06:18:03.160616","last_peered":"2020-03-28 06:18:03.160616","last_clean":"2020-03-28 06:18:03.160616","last_became_active":"2020-03-28 06:18:03.113040","last_became_peered":"2020-03-28 06:18:03.113040","last_unstale":"2020-03-28 06:18:03.160616","last_undegraded":"2020-03-28 06:18:03.160616","last_fullsized":"2020-03-28 06:18:03.160616","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.143144","last_change":"2020-03-28 06:18:03.105635","last_active":"2020-03-28 06:18:03.143144","last_peered":"2020-03-28 06:18:03.143144","last_clean":"2020-03-28 06:18:03.143144","last_became_active":"2020-03-28 06:18:03.072615","last_became_peered":"2020-03-28 06:18:03.072615","last_unstale":"2020-03-28 06:18:03.143144","last_undegraded":"2020-03-28 06:18:03.143144","last_fullsized":"2020-03-28 06:18:03.143144","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.157288","last_change":"2020-03-28 06:18:03.111593","last_active":"2020-03-28 06:18:03.157288","last_peered":"2020-03-28 06:18:03.157288","last_clean":"2020-03-28 06:18:03.157288","last_became_active":"2020-03-28 06:18:03.103125","last_became_peered":"2020-03-28 06:18:03.103125","last_unstale":"2020-03-28 06:18:03.157288","last_undegraded":"2020-03-28 06:18:03.157288","last_fullsized":"2020-03-28 06:18:03.157288","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.155371","last_change":"2020-03-28 06:18:03.119404","last_active":"2020-03-28 06:18:03.155371","last_peered":"2020-03-28 06:18:03.155371","last_clean":"2020-03-28 06:18:03.155371","last_became_active":"2020-03-28 06:18:03.100376","last_became_peered":"2020-03-28 06:18:03.100376","last_unstale":"2020-03-28 06:18:03.155371","last_undegraded":"2020-03-28 06:18:03.155371","last_fullsized":"2020-03-28 06:18:03.155371","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.151416","last_change":"2020-03-28 06:18:03.111003","last_active":"2020-03-28 06:18:03.151416","last_peered":"2020-03-28 06:18:03.151416","last_clean":"2020-03-28 06:18:03.151416","last_became_active":"2020-03-28 06:18:03.050457","last_became_peered":"2020-03-28 06:18:03.050457","last_unstale":"2020-03-28 06:18:03.151416","last_undegraded":"2020-03-28 06:18:03.151416","last_fullsized":"2020-03-28 06:18:03.151416","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:02.941750","last_change":"2020-03-28 06:18:02.146544","last_active":"2020-03-28 06:18:02.941750","last_peered":"2020-03-28 06:18:02.941750","last_clean":"2020-03-28 06:18:02.941750","last_became_active":"2020-03-28 06:18:02.092388","last_became_peered":"2020-03-28 06:18:02.092388","last_unstale":"2020-03-28 06:18:02.941750","last_undegraded":"2020-03-28 06:18:02.941750","last_fullsized":"2020-03-28 06:18:02.941750","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.157931","last_change":"2020-03-28 06:18:03.052085","last_active":"2020-03-28 06:18:03.157931","last_peered":"2020-03-28 06:18:03.157931","last_clean":"2020-03-28 06:18:03.157931","last_became_active":"2020-03-28 06:18:03.003911","last_became_peered":"2020-03-28 06:18:03.003911","last_unstale":"2020-03-28 06:18:03.157931","last_undegraded":"2020-03-28 06:18:03.157931","last_fullsized":"2020-03-28 06:18:03.157931","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-03-28T06:18:13.294 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 pg dump --format=json 2020-03-28T06:18:14.281 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:18:14.282 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2020-03-28T06:18:14.303 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2020-03-28 06:18:12.339016","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":86703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.159513","last_change":"2020-03-28 06:18:03.079635","last_active":"2020-03-28 06:18:03.159513","last_peered":"2020-03-28 06:18:03.159513","last_clean":"2020-03-28 06:18:03.159513","last_became_active":"2020-03-28 06:18:03.063900","last_became_peered":"2020-03-28 06:18:03.063900","last_unstale":"2020-03-28 06:18:03.159513","last_undegraded":"2020-03-28 06:18:03.159513","last_fullsized":"2020-03-28 06:18:03.159513","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.160616","last_change":"2020-03-28 06:18:03.146905","last_active":"2020-03-28 06:18:03.160616","last_peered":"2020-03-28 06:18:03.160616","last_clean":"2020-03-28 06:18:03.160616","last_became_active":"2020-03-28 06:18:03.113040","last_became_peered":"2020-03-28 06:18:03.113040","last_unstale":"2020-03-28 06:18:03.160616","last_undegraded":"2020-03-28 06:18:03.160616","last_fullsized":"2020-03-28 06:18:03.160616","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.143144","last_change":"2020-03-28 06:18:03.105635","last_active":"2020-03-28 06:18:03.143144","last_peered":"2020-03-28 06:18:03.143144","last_clean":"2020-03-28 06:18:03.143144","last_became_active":"2020-03-28 06:18:03.072615","last_became_peered":"2020-03-28 06:18:03.072615","last_unstale":"2020-03-28 06:18:03.143144","last_undegraded":"2020-03-28 06:18:03.143144","last_fullsized":"2020-03-28 06:18:03.143144","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.157288","last_change":"2020-03-28 06:18:03.111593","last_active":"2020-03-28 06:18:03.157288","last_peered":"2020-03-28 06:18:03.157288","last_clean":"2020-03-28 06:18:03.157288","last_became_active":"2020-03-28 06:18:03.103125","last_became_peered":"2020-03-28 06:18:03.103125","last_unstale":"2020-03-28 06:18:03.157288","last_undegraded":"2020-03-28 06:18:03.157288","last_fullsized":"2020-03-28 06:18:03.157288","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.155371","last_change":"2020-03-28 06:18:03.119404","last_active":"2020-03-28 06:18:03.155371","last_peered":"2020-03-28 06:18:03.155371","last_clean":"2020-03-28 06:18:03.155371","last_became_active":"2020-03-28 06:18:03.100376","last_became_peered":"2020-03-28 06:18:03.100376","last_unstale":"2020-03-28 06:18:03.155371","last_undegraded":"2020-03-28 06:18:03.155371","last_fullsized":"2020-03-28 06:18:03.155371","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.151416","last_change":"2020-03-28 06:18:03.111003","last_active":"2020-03-28 06:18:03.151416","last_peered":"2020-03-28 06:18:03.151416","last_clean":"2020-03-28 06:18:03.151416","last_became_active":"2020-03-28 06:18:03.050457","last_became_peered":"2020-03-28 06:18:03.050457","last_unstale":"2020-03-28 06:18:03.151416","last_undegraded":"2020-03-28 06:18:03.151416","last_fullsized":"2020-03-28 06:18:03.151416","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:02.941750","last_change":"2020-03-28 06:18:02.146544","last_active":"2020-03-28 06:18:02.941750","last_peered":"2020-03-28 06:18:02.941750","last_clean":"2020-03-28 06:18:02.941750","last_became_active":"2020-03-28 06:18:02.092388","last_became_peered":"2020-03-28 06:18:02.092388","last_unstale":"2020-03-28 06:18:02.941750","last_undegraded":"2020-03-28 06:18:02.941750","last_fullsized":"2020-03-28 06:18:02.941750","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:18:03.157931","last_change":"2020-03-28 06:18:03.052085","last_active":"2020-03-28 06:18:03.157931","last_peered":"2020-03-28 06:18:03.157931","last_clean":"2020-03-28 06:18:03.157931","last_became_active":"2020-03-28 06:18:03.003911","last_became_peered":"2020-03-28 06:18:03.003911","last_unstale":"2020-03-28 06:18:03.157931","last_undegraded":"2020-03-28 06:18:03.157931","last_fullsized":"2020-03-28 06:18:03.157931","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:18:00.461819","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:18:00.461819","last_clean_scrub_stamp":"2020-03-28 06:18:00.461819","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-03-28T06:18:14.305 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 status --format=json-pretty 2020-03-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi156.stdout: 2020-03-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2020-03-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi156.stdout: "fsid": "ff1cba1e-a6a7-4868-8184-daa87358bc60", 2020-03-28T06:18:15.337 INFO:teuthology.orchestra.run.smithi156.stdout: "health": { 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: "checks": {}, 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: "status": "HEALTH_OK" 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: "election_epoch": 3, 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: "quorum": [ 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: 0 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.338 INFO:teuthology.orchestra.run.smithi156.stdout: "quorum_names": [ 2020-03-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi156.stdout: "a" 2020-03-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi156.stdout: "quorum_age": 39, 2020-03-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi156.stdout: "monmap": { 2020-03-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 1, 2020-03-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi156.stdout: "fsid": "ff1cba1e-a6a7-4868-8184-daa87358bc60", 2020-03-28T06:18:15.339 INFO:teuthology.orchestra.run.smithi156.stdout: "modified": "2020-03-28 06:17:12.323554", 2020-03-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi156.stdout: "created": "2020-03-28 06:17:12.323554", 2020-03-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi156.stdout: "min_mon_release": 14, 2020-03-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi156.stdout: "min_mon_release_name": "nautilus", 2020-03-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi156.stdout: "features": { 2020-03-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi156.stdout: "persistent": [ 2020-03-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi156.stdout: "kraken", 2020-03-28T06:18:15.340 INFO:teuthology.orchestra.run.smithi156.stdout: "luminous", 2020-03-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi156.stdout: "mimic", 2020-03-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap-prune", 2020-03-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi156.stdout: "nautilus" 2020-03-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi156.stdout: "optional": [] 2020-03-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.341 INFO:teuthology.orchestra.run.smithi156.stdout: "mons": [ 2020-03-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi156.stdout: "rank": 0, 2020-03-28T06:18:15.342 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "a", 2020-03-28T06:18:15.343 INFO:teuthology.orchestra.run.smithi156.stdout: "public_addrs": { 2020-03-28T06:18:15.343 INFO:teuthology.orchestra.run.smithi156.stdout: "addrvec": [ 2020-03-28T06:18:15.343 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.343 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v2", 2020-03-28T06:18:15.343 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:3300", 2020-03-28T06:18:15.344 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 0 2020-03-28T06:18:15.344 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.344 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.344 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2020-03-28T06:18:15.344 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:6789", 2020-03-28T06:18:15.344 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 0 2020-03-28T06:18:15.344 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2020-03-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:6789/0", 2020-03-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi156.stdout: "public_addr": "172.21.15.156:6789/0" 2020-03-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2020-03-28T06:18:15.345 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap": { 2020-03-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap": { 2020-03-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 13, 2020-03-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi156.stdout: "num_osds": 3, 2020-03-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi156.stdout: "num_up_osds": 3, 2020-03-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi156.stdout: "num_in_osds": 3, 2020-03-28T06:18:15.346 INFO:teuthology.orchestra.run.smithi156.stdout: "num_remapped_pgs": 0 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap": { 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: "pgs_by_state": [ 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: "state_name": "active+clean", 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: "count": 8 2020-03-28T06:18:15.347 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: "num_pgs": 8, 2020-03-28T06:18:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: "num_pools": 1, 2020-03-28T06:18:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: "num_objects": 0, 2020-03-28T06:18:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: "data_bytes": 0, 2020-03-28T06:18:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: "bytes_used": 3221594112, 2020-03-28T06:18:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: "bytes_avail": 318900953088, 2020-03-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi156.stdout: "bytes_total": 322122547200 2020-03-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi156.stdout: "fsmap": { 2020-03-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 1, 2020-03-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi156.stdout: "by_rank": [], 2020-03-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi156.stdout: "up:standby": 0 2020-03-28T06:18:15.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.350 INFO:teuthology.orchestra.run.smithi156.stdout: "mgrmap": { 2020-03-28T06:18:15.350 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 3, 2020-03-28T06:18:15.350 INFO:teuthology.orchestra.run.smithi156.stdout: "active_gid": 4102, 2020-03-28T06:18:15.350 INFO:teuthology.orchestra.run.smithi156.stdout: "active_name": "x", 2020-03-28T06:18:15.350 INFO:teuthology.orchestra.run.smithi156.stdout: "active_addrs": { 2020-03-28T06:18:15.350 INFO:teuthology.orchestra.run.smithi156.stdout: "addrvec": [ 2020-03-28T06:18:15.350 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.351 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v2", 2020-03-28T06:18:15.351 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:6800", 2020-03-28T06:18:15.351 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 8059 2020-03-28T06:18:15.351 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.351 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.351 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2020-03-28T06:18:15.351 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "172.21.15.156:6801", 2020-03-28T06:18:15.352 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 8059 2020-03-28T06:18:15.352 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.352 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2020-03-28T06:18:15.352 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.352 INFO:teuthology.orchestra.run.smithi156.stdout: "active_addr": "172.21.15.156:6801/8059", 2020-03-28T06:18:15.352 INFO:teuthology.orchestra.run.smithi156.stdout: "active_change": "2020-03-28 06:17:40.245487", 2020-03-28T06:18:15.353 INFO:teuthology.orchestra.run.smithi156.stdout: "available": true, 2020-03-28T06:18:15.353 INFO:teuthology.orchestra.run.smithi156.stdout: "standbys": [], 2020-03-28T06:18:15.353 INFO:teuthology.orchestra.run.smithi156.stdout: "modules": [ 2020-03-28T06:18:15.354 INFO:teuthology.orchestra.run.smithi156.stdout: "iostat", 2020-03-28T06:18:15.354 INFO:teuthology.orchestra.run.smithi156.stdout: "restful" 2020-03-28T06:18:15.354 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.354 INFO:teuthology.orchestra.run.smithi156.stdout: "available_modules": [ 2020-03-28T06:18:15.354 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.354 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "alerts", 2020-03-28T06:18:15.354 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "interval": { 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "interval", 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.355 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "60", 2020-03-28T06:18:15.356 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.356 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.356 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.356 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "How frequently to reexamine health status", 2020-03-28T06:18:15.356 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.356 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.356 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_destination": { 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_destination", 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.357 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Email address to send alerts to", 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.358 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_from_name": { 2020-03-28T06:18:15.359 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_from_name", 2020-03-28T06:18:15.359 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.359 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.359 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.359 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "Ceph", 2020-03-28T06:18:15.359 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.359 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Email From: name", 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_host": { 2020-03-28T06:18:15.360 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_host", 2020-03-28T06:18:15.361 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.361 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.361 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.361 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.361 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.361 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.361 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.362 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "SMTP server", 2020-03-28T06:18:15.362 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.362 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.362 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.362 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.363 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_password": { 2020-03-28T06:18:15.363 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_password", 2020-03-28T06:18:15.363 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.364 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.364 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.364 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.364 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.364 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.364 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.364 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Password to authenticate with", 2020-03-28T06:18:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_port": { 2020-03-28T06:18:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_port", 2020-03-28T06:18:15.365 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "465", 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "SMTP port", 2020-03-28T06:18:15.366 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.367 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.367 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.367 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.367 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_sender": { 2020-03-28T06:18:15.367 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_sender", 2020-03-28T06:18:15.367 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.367 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.368 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.368 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.368 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.368 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.368 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.368 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "SMTP envelope sender", 2020-03-28T06:18:15.368 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.369 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.369 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.369 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.369 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_ssl": { 2020-03-28T06:18:15.369 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_ssl", 2020-03-28T06:18:15.369 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.369 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Use SSL to connect to SMTP server", 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.371 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.371 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.371 INFO:teuthology.orchestra.run.smithi156.stdout: "smtp_user": { 2020-03-28T06:18:15.371 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "smtp_user", 2020-03-28T06:18:15.371 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.371 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.371 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "User to authenticate as", 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.372 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.373 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.373 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.373 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.374 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.374 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ansible", 2020-03-28T06:18:15.374 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.374 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.374 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.375 INFO:teuthology.orchestra.run.smithi156.stdout: "password": { 2020-03-28T06:18:15.375 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "password", 2020-03-28T06:18:15.375 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.375 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.375 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.375 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.375 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.376 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.376 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.376 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.376 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.376 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.376 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.376 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "server_url": { 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "server_url", 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.377 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.378 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.378 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.378 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.378 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.378 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.378 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.378 INFO:teuthology.orchestra.run.smithi156.stdout: "username": { 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "username", 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.379 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.380 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.380 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.380 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.380 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.380 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.380 INFO:teuthology.orchestra.run.smithi156.stdout: "verify_server": { 2020-03-28T06:18:15.380 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "verify_server", 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.381 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.382 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.382 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.382 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.382 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.382 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.382 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.382 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.383 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "balancer", 2020-03-28T06:18:15.383 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.383 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.383 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.383 INFO:teuthology.orchestra.run.smithi156.stdout: "active": { 2020-03-28T06:18:15.383 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "active", 2020-03-28T06:18:15.384 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.384 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.384 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.385 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.385 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.385 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.385 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.385 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "automatically balance PGs across cluster", 2020-03-28T06:18:15.385 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.385 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.386 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.386 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.386 INFO:teuthology.orchestra.run.smithi156.stdout: "begin_time": { 2020-03-28T06:18:15.386 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "begin_time", 2020-03-28T06:18:15.386 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.386 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.386 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.387 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "0000", 2020-03-28T06:18:15.387 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.387 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.387 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.387 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "beginning time of day to automatically balance", 2020-03-28T06:18:15.387 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "This is a time of day in the format HHMM.", 2020-03-28T06:18:15.387 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.388 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.388 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.388 INFO:teuthology.orchestra.run.smithi156.stdout: "begin_weekday": { 2020-03-28T06:18:15.388 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "begin_weekday", 2020-03-28T06:18:15.388 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "uint", 2020-03-28T06:18:15.388 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.388 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "0", 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "0", 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "7", 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.389 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.390 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.390 INFO:teuthology.orchestra.run.smithi156.stdout: "crush_compat_max_iterations": { 2020-03-28T06:18:15.390 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "crush_compat_max_iterations", 2020-03-28T06:18:15.390 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "uint", 2020-03-28T06:18:15.390 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.390 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.390 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "25", 2020-03-28T06:18:15.391 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "1", 2020-03-28T06:18:15.391 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "250", 2020-03-28T06:18:15.391 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.391 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "maximum number of iterations to attempt optimization", 2020-03-28T06:18:15.391 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.391 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.391 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.392 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.392 INFO:teuthology.orchestra.run.smithi156.stdout: "crush_compat_metrics": { 2020-03-28T06:18:15.392 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "crush_compat_metrics", 2020-03-28T06:18:15.392 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.392 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.392 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.392 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "pgs,objects,bytes", 2020-03-28T06:18:15.393 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.393 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.393 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.393 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "metrics with which to calculate OSD utilization", 2020-03-28T06:18:15.393 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2020-03-28T06:18:15.393 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.394 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.394 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.394 INFO:teuthology.orchestra.run.smithi156.stdout: "crush_compat_step": { 2020-03-28T06:18:15.395 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "crush_compat_step", 2020-03-28T06:18:15.395 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "float", 2020-03-28T06:18:15.395 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.395 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.395 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "0.5", 2020-03-28T06:18:15.395 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "0.001", 2020-03-28T06:18:15.395 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "0.999", 2020-03-28T06:18:15.396 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.396 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "aggressiveness of optimization", 2020-03-28T06:18:15.396 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2020-03-28T06:18:15.396 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.396 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.396 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.396 INFO:teuthology.orchestra.run.smithi156.stdout: "end_time": { 2020-03-28T06:18:15.397 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "end_time", 2020-03-28T06:18:15.397 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.397 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.397 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.397 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "2400", 2020-03-28T06:18:15.397 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.397 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "ending time of day to automatically balance", 2020-03-28T06:18:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "This is a time of day in the format HHMM.", 2020-03-28T06:18:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: "end_weekday": { 2020-03-28T06:18:15.399 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "end_weekday", 2020-03-28T06:18:15.399 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "uint", 2020-03-28T06:18:15.399 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.399 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.399 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "7", 2020-03-28T06:18:15.399 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "0", 2020-03-28T06:18:15.399 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "7", 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: "min_score": { 2020-03-28T06:18:15.400 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "min_score", 2020-03-28T06:18:15.401 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "float", 2020-03-28T06:18:15.401 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.401 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.401 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "0", 2020-03-28T06:18:15.401 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.401 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.401 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.402 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "minimum score, below which no optimization is attempted", 2020-03-28T06:18:15.402 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.402 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.402 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.402 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.402 INFO:teuthology.orchestra.run.smithi156.stdout: "mode": { 2020-03-28T06:18:15.402 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "mode", 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "none", 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [ 2020-03-28T06:18:15.403 INFO:teuthology.orchestra.run.smithi156.stdout: "crush-compat", 2020-03-28T06:18:15.404 INFO:teuthology.orchestra.run.smithi156.stdout: "none", 2020-03-28T06:18:15.404 INFO:teuthology.orchestra.run.smithi156.stdout: "upmap" 2020-03-28T06:18:15.405 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.405 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Balancer mode", 2020-03-28T06:18:15.405 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.405 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.405 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: "pool_ids": { 2020-03-28T06:18:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "pool_ids", 2020-03-28T06:18:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "pools which the automatic balancing will be limited to", 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.408 INFO:teuthology.orchestra.run.smithi156.stdout: "sleep_interval": { 2020-03-28T06:18:15.408 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "sleep_interval", 2020-03-28T06:18:15.408 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.408 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.408 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.408 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "60", 2020-03-28T06:18:15.408 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "how frequently to wake up and attempt optimization", 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.409 INFO:teuthology.orchestra.run.smithi156.stdout: "upmap_max_deviation": { 2020-03-28T06:18:15.410 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "upmap_max_deviation", 2020-03-28T06:18:15.410 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.410 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.410 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.410 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "5", 2020-03-28T06:18:15.410 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "1", 2020-03-28T06:18:15.410 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.411 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.411 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "deviation below which no optimization is attempted", 2020-03-28T06:18:15.411 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2020-03-28T06:18:15.411 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.411 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.411 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.411 INFO:teuthology.orchestra.run.smithi156.stdout: "upmap_max_iterations": { 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "upmap_max_iterations", 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "uint", 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "10", 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.412 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.413 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "maximum upmap optimization iterations", 2020-03-28T06:18:15.413 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.413 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.413 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.413 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.413 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.413 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.414 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.414 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "crash", 2020-03-28T06:18:15.414 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.414 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.414 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.415 INFO:teuthology.orchestra.run.smithi156.stdout: "retain_interval": { 2020-03-28T06:18:15.415 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "retain_interval", 2020-03-28T06:18:15.415 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.416 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.416 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.416 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "31536000", 2020-03-28T06:18:15.416 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.416 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.416 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.416 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "how long to retain crashes before pruning them", 2020-03-28T06:18:15.417 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.417 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.417 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.417 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.417 INFO:teuthology.orchestra.run.smithi156.stdout: "warn_recent_interval": { 2020-03-28T06:18:15.417 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "warn_recent_interval", 2020-03-28T06:18:15.417 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "1209600", 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "time interval in which to warn about recent crashes", 2020-03-28T06:18:15.418 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.419 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.419 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.419 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.419 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.419 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.419 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.420 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "dashboard", 2020-03-28T06:18:15.420 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.420 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.420 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.421 INFO:teuthology.orchestra.run.smithi156.stdout: "ALERTMANAGER_API_HOST": { 2020-03-28T06:18:15.421 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ALERTMANAGER_API_HOST", 2020-03-28T06:18:15.421 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.421 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.421 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.421 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.421 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.422 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.422 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.422 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.422 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.422 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.422 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.422 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.423 INFO:teuthology.orchestra.run.smithi156.stdout: "AUDIT_API_ENABLED": { 2020-03-28T06:18:15.423 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "AUDIT_API_ENABLED", 2020-03-28T06:18:15.423 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.423 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.423 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.423 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.423 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.424 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.425 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: "AUDIT_API_LOG_PAYLOAD": { 2020-03-28T06:18:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2020-03-28T06:18:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.426 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.427 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.428 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.428 INFO:teuthology.orchestra.run.smithi156.stdout: "ENABLE_BROWSABLE_API": { 2020-03-28T06:18:15.428 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ENABLE_BROWSABLE_API", 2020-03-28T06:18:15.428 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.428 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.428 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.428 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.429 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.430 INFO:teuthology.orchestra.run.smithi156.stdout: "FEATURE_TOGGLE_cephfs": { 2020-03-28T06:18:15.430 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "FEATURE_TOGGLE_cephfs", 2020-03-28T06:18:15.430 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.430 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.430 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.430 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.430 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.431 INFO:teuthology.orchestra.run.smithi156.stdout: "FEATURE_TOGGLE_iscsi": { 2020-03-28T06:18:15.432 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "FEATURE_TOGGLE_iscsi", 2020-03-28T06:18:15.432 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.432 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.432 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.432 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.432 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.432 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.433 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.433 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.433 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.433 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.433 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.433 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.433 INFO:teuthology.orchestra.run.smithi156.stdout: "FEATURE_TOGGLE_mirroring": { 2020-03-28T06:18:15.434 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "FEATURE_TOGGLE_mirroring", 2020-03-28T06:18:15.434 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.434 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.434 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.434 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.434 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.434 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.435 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.435 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.435 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.436 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.436 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.436 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.436 INFO:teuthology.orchestra.run.smithi156.stdout: "FEATURE_TOGGLE_rbd": { 2020-03-28T06:18:15.436 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "FEATURE_TOGGLE_rbd", 2020-03-28T06:18:15.436 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.437 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.437 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.437 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.437 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.437 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.437 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.437 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: "FEATURE_TOGGLE_rgw": { 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "FEATURE_TOGGLE_rgw", 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.438 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.439 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.440 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.440 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.440 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.440 INFO:teuthology.orchestra.run.smithi156.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2020-03-28T06:18:15.440 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2020-03-28T06:18:15.440 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.440 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.441 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.442 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.442 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.442 INFO:teuthology.orchestra.run.smithi156.stdout: "GRAFANA_API_PASSWORD": { 2020-03-28T06:18:15.442 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "GRAFANA_API_PASSWORD", 2020-03-28T06:18:15.442 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.442 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.442 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "admin", 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.443 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.444 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.444 INFO:teuthology.orchestra.run.smithi156.stdout: "GRAFANA_API_SSL_VERIFY": { 2020-03-28T06:18:15.444 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2020-03-28T06:18:15.444 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.444 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.444 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.444 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.445 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.445 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.445 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.445 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.446 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.446 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.446 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: "GRAFANA_API_URL": { 2020-03-28T06:18:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "GRAFANA_API_URL", 2020-03-28T06:18:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.447 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.448 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.449 INFO:teuthology.orchestra.run.smithi156.stdout: "GRAFANA_API_USERNAME": { 2020-03-28T06:18:15.449 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "GRAFANA_API_USERNAME", 2020-03-28T06:18:15.449 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.449 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.449 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.449 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "admin", 2020-03-28T06:18:15.449 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.450 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.450 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.450 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.450 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.450 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.450 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.451 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.452 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.452 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.452 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.452 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.452 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.452 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.452 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.453 INFO:teuthology.orchestra.run.smithi156.stdout: "ISCSI_API_SSL_VERIFICATION": { 2020-03-28T06:18:15.453 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2020-03-28T06:18:15.453 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.453 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.453 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.453 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.453 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: "PROMETHEUS_API_HOST": { 2020-03-28T06:18:15.455 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "PROMETHEUS_API_HOST", 2020-03-28T06:18:15.455 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.455 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.455 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.455 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.456 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.456 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.456 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: "REST_REQUESTS_TIMEOUT": { 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "REST_REQUESTS_TIMEOUT", 2020-03-28T06:18:15.457 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.458 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.458 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.458 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "45", 2020-03-28T06:18:15.458 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.458 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.458 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.458 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.459 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.459 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.459 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.459 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.459 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_ACCESS_KEY": { 2020-03-28T06:18:15.459 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_ACCESS_KEY", 2020-03-28T06:18:15.459 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.460 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.460 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.460 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.460 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.460 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.460 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.460 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_ADMIN_RESOURCE": { 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.461 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.462 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.462 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "admin", 2020-03-28T06:18:15.462 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.462 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.462 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.462 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.462 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_HOST": { 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_HOST", 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.463 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.465 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.465 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.465 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_PORT": { 2020-03-28T06:18:15.465 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_PORT", 2020-03-28T06:18:15.465 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.465 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.466 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.466 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "80", 2020-03-28T06:18:15.466 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.466 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.467 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.467 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.467 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.467 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.467 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.467 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_SCHEME": { 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_SCHEME", 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "http", 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.468 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.469 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_SECRET_KEY": { 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_SECRET_KEY", 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.470 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.471 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.471 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.471 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.471 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.471 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.471 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_SSL_VERIFY": { 2020-03-28T06:18:15.471 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_SSL_VERIFY", 2020-03-28T06:18:15.472 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.472 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.472 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.472 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.472 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.472 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.472 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: "RGW_API_USER_ID": { 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "RGW_API_USER_ID", 2020-03-28T06:18:15.473 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.474 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: "crt_file": { 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "crt_file", 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.475 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.476 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.476 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.476 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.476 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.477 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.477 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.477 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.477 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.478 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.478 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.478 INFO:teuthology.orchestra.run.smithi156.stdout: "debug": { 2020-03-28T06:18:15.478 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "debug", 2020-03-28T06:18:15.478 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.478 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.478 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Enable/disable debug options", 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.479 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.480 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.480 INFO:teuthology.orchestra.run.smithi156.stdout: "jwt_token_ttl": { 2020-03-28T06:18:15.480 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "jwt_token_ttl", 2020-03-28T06:18:15.480 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.480 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.480 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.480 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "28800", 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.481 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.482 INFO:teuthology.orchestra.run.smithi156.stdout: "key_file": { 2020-03-28T06:18:15.482 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "key_file", 2020-03-28T06:18:15.482 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.482 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.482 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.482 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.482 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.483 INFO:teuthology.orchestra.run.smithi156.stdout: "password": { 2020-03-28T06:18:15.484 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "password", 2020-03-28T06:18:15.484 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.484 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.484 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.484 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.484 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.484 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.485 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.485 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.485 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.485 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.485 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.485 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.485 INFO:teuthology.orchestra.run.smithi156.stdout: "server_addr": { 2020-03-28T06:18:15.486 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "server_addr", 2020-03-28T06:18:15.486 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.486 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.486 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.486 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "::", 2020-03-28T06:18:15.486 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.487 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.487 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.487 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.488 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.488 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.488 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.488 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.488 INFO:teuthology.orchestra.run.smithi156.stdout: "server_port": { 2020-03-28T06:18:15.488 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "server_port", 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "8080", 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.489 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.490 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.490 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.490 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.490 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.490 INFO:teuthology.orchestra.run.smithi156.stdout: "ssl": { 2020-03-28T06:18:15.490 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ssl", 2020-03-28T06:18:15.490 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.491 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.492 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.492 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.492 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.492 INFO:teuthology.orchestra.run.smithi156.stdout: "ssl_server_port": { 2020-03-28T06:18:15.492 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ssl_server_port", 2020-03-28T06:18:15.492 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.492 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "8443", 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.493 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.494 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.494 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.494 INFO:teuthology.orchestra.run.smithi156.stdout: "standby_behaviour": { 2020-03-28T06:18:15.494 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "standby_behaviour", 2020-03-28T06:18:15.494 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.494 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.494 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "redirect", 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [ 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: "error", 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: "redirect" 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.495 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.496 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.496 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.496 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.496 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.496 INFO:teuthology.orchestra.run.smithi156.stdout: "standby_error_status_code": { 2020-03-28T06:18:15.497 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "standby_error_status_code", 2020-03-28T06:18:15.497 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.497 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.497 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.498 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "500", 2020-03-28T06:18:15.498 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "400", 2020-03-28T06:18:15.498 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "599", 2020-03-28T06:18:15.498 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.498 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.498 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.499 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.499 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.499 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.499 INFO:teuthology.orchestra.run.smithi156.stdout: "url_prefix": { 2020-03-28T06:18:15.499 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "url_prefix", 2020-03-28T06:18:15.499 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.500 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.500 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.500 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.500 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.500 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.500 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.500 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.501 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.501 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.501 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.501 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.501 INFO:teuthology.orchestra.run.smithi156.stdout: "username": { 2020-03-28T06:18:15.501 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "username", 2020-03-28T06:18:15.501 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.502 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "deepsea", 2020-03-28T06:18:15.503 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.504 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.504 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.504 INFO:teuthology.orchestra.run.smithi156.stdout: "salt_api_eauth": { 2020-03-28T06:18:15.504 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "salt_api_eauth", 2020-03-28T06:18:15.504 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.504 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.504 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "sharedsecret", 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.505 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.506 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.506 INFO:teuthology.orchestra.run.smithi156.stdout: "salt_api_password": { 2020-03-28T06:18:15.506 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "salt_api_password", 2020-03-28T06:18:15.506 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.506 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.506 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.507 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.508 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.508 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.508 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.508 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.508 INFO:teuthology.orchestra.run.smithi156.stdout: "salt_api_url": { 2020-03-28T06:18:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "salt_api_url", 2020-03-28T06:18:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: "salt_api_username": { 2020-03-28T06:18:15.510 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "salt_api_username", 2020-03-28T06:18:15.511 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.511 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.511 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.511 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.511 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.511 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.511 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.512 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.513 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "devicehealth", 2020-03-28T06:18:15.513 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.513 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.513 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.513 INFO:teuthology.orchestra.run.smithi156.stdout: "enable_monitoring": { 2020-03-28T06:18:15.513 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "enable_monitoring", 2020-03-28T06:18:15.513 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "monitor device health metrics", 2020-03-28T06:18:15.514 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.515 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.515 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.515 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.515 INFO:teuthology.orchestra.run.smithi156.stdout: "mark_out_threshold": { 2020-03-28T06:18:15.515 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "mark_out_threshold", 2020-03-28T06:18:15.515 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.515 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "2419200", 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "automatically mark OSD if it may fail before this long", 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.516 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.517 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.517 INFO:teuthology.orchestra.run.smithi156.stdout: "pool_name": { 2020-03-28T06:18:15.517 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "pool_name", 2020-03-28T06:18:15.517 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.518 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.518 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.518 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "device_health_metrics", 2020-03-28T06:18:15.518 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "name of pool in which to store device health metrics", 2020-03-28T06:18:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.520 INFO:teuthology.orchestra.run.smithi156.stdout: "retention_period": { 2020-03-28T06:18:15.520 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "retention_period", 2020-03-28T06:18:15.520 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.520 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.520 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.521 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "15552000", 2020-03-28T06:18:15.521 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.521 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.521 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.521 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "how long to retain device health metrics", 2020-03-28T06:18:15.521 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.521 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: "scrape_frequency": { 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "scrape_frequency", 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.522 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "86400", 2020-03-28T06:18:15.523 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.523 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.523 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.523 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "how frequently to scrape device health metrics", 2020-03-28T06:18:15.523 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.523 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.523 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: "self_heal": { 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "self_heal", 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.524 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.525 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.525 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.525 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "preemptively heal cluster around devices that may fail", 2020-03-28T06:18:15.525 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.525 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.525 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.525 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "sleep_interval": { 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "sleep_interval", 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "600", 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.526 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.527 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.527 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "how frequently to wake up and check device health", 2020-03-28T06:18:15.527 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.527 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.528 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.528 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.528 INFO:teuthology.orchestra.run.smithi156.stdout: "warn_threshold": { 2020-03-28T06:18:15.529 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "warn_threshold", 2020-03-28T06:18:15.529 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.529 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.529 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.529 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "7257600", 2020-03-28T06:18:15.529 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.529 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.530 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.530 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "raise health warning if OSD may fail before this long", 2020-03-28T06:18:15.531 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.531 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.531 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.532 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.532 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.532 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.532 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.532 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_cloud", 2020-03-28T06:18:15.532 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.532 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_cert_context": { 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_cert_context", 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.533 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.534 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.534 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.534 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.534 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.534 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.534 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.534 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_default_authority": { 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_default_authority", 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "localhost", 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.535 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.536 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.536 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.536 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.536 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.536 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.536 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_password": { 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_password", 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.537 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.538 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.538 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.538 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.538 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.538 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.538 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.538 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_port": { 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_port", 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "31400", 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.539 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.540 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.540 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.540 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.540 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.540 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.540 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_retrieve_prediction_interval": { 2020-03-28T06:18:15.540 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_retrieve_prediction_interval", 2020-03-28T06:18:15.541 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.541 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.541 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.541 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "43200", 2020-03-28T06:18:15.541 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.541 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.541 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_server": { 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_server", 2020-03-28T06:18:15.542 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.543 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.543 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.543 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.543 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.543 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.543 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.543 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_ssl_target_name_override": { 2020-03-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_ssl_target_name_override", 2020-03-28T06:18:15.544 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.545 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.545 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.545 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "localhost", 2020-03-28T06:18:15.545 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.545 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.545 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.545 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_upload_metrics_interval": { 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_upload_metrics_interval", 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.546 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.547 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.547 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "600", 2020-03-28T06:18:15.547 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.547 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.547 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.547 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.547 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_upload_smart_interval": { 2020-03-28T06:18:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_upload_smart_interval", 2020-03-28T06:18:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "43200", 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.549 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.550 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.550 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.550 INFO:teuthology.orchestra.run.smithi156.stdout: "diskprediction_user": { 2020-03-28T06:18:15.550 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_user", 2020-03-28T06:18:15.550 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.550 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.550 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.551 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.551 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.551 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.551 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.551 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.551 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.551 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.552 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.552 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.552 INFO:teuthology.orchestra.run.smithi156.stdout: "sleep_interval": { 2020-03-28T06:18:15.552 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "sleep_interval", 2020-03-28T06:18:15.552 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.552 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.552 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "600", 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.553 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.554 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.554 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.554 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.554 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.554 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "diskprediction_local", 2020-03-28T06:18:15.554 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.554 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "predict_interval": { 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "predict_interval", 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "86400", 2020-03-28T06:18:15.555 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.556 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.556 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.556 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.556 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.556 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.556 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.556 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.557 INFO:teuthology.orchestra.run.smithi156.stdout: "sleep_interval": { 2020-03-28T06:18:15.557 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "sleep_interval", 2020-03-28T06:18:15.557 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.557 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.557 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.557 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "600", 2020-03-28T06:18:15.557 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.558 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.559 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.559 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.559 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "influx", 2020-03-28T06:18:15.559 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": false, 2020-03-28T06:18:15.559 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "influxdb python module not found", 2020-03-28T06:18:15.559 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.559 INFO:teuthology.orchestra.run.smithi156.stdout: "batch_size": { 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "batch_size", 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "5000", 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.560 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.561 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.561 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.561 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.561 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.561 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.561 INFO:teuthology.orchestra.run.smithi156.stdout: "database": { 2020-03-28T06:18:15.561 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "database", 2020-03-28T06:18:15.562 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.562 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.562 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.562 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "ceph", 2020-03-28T06:18:15.562 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.562 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.562 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": { 2020-03-28T06:18:15.563 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "hostname", 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.564 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.565 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.565 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.565 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.565 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.565 INFO:teuthology.orchestra.run.smithi156.stdout: "interval": { 2020-03-28T06:18:15.565 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "interval", 2020-03-28T06:18:15.565 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "30", 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.566 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.567 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.567 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.567 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.567 INFO:teuthology.orchestra.run.smithi156.stdout: "password": { 2020-03-28T06:18:15.567 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "password", 2020-03-28T06:18:15.567 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.567 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.568 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi156.stdout: "port": { 2020-03-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "port", 2020-03-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.569 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "8086", 2020-03-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.570 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: "ssl": { 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ssl", 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.571 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "false", 2020-03-28T06:18:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.572 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.573 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.573 INFO:teuthology.orchestra.run.smithi156.stdout: "threads": { 2020-03-28T06:18:15.573 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "threads", 2020-03-28T06:18:15.573 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.573 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.573 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.573 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "5", 2020-03-28T06:18:15.574 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.574 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.574 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.574 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.574 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.574 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.574 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: "username": { 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "username", 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.575 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.576 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.576 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.576 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.576 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.576 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.576 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.576 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "verify_ssl": { 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "verify_ssl", 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "true", 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.577 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.578 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.578 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.578 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.578 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.578 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.578 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.578 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.579 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.579 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.579 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "insights", 2020-03-28T06:18:15.579 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.579 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.579 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": {} 2020-03-28T06:18:15.579 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.580 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.580 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "iostat", 2020-03-28T06:18:15.580 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.580 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.580 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": {} 2020-03-28T06:18:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "localpool", 2020-03-28T06:18:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.581 INFO:teuthology.orchestra.run.smithi156.stdout: "failure_domain": { 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "failure_domain", 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "host", 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.582 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "failure domain for any created local pool", 2020-03-28T06:18:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: "min_size": { 2020-03-28T06:18:15.583 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "min_size", 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.584 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "default min_size for any created local pool", 2020-03-28T06:18:15.585 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.585 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.585 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.585 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.585 INFO:teuthology.orchestra.run.smithi156.stdout: "num_rep": { 2020-03-28T06:18:15.585 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "num_rep", 2020-03-28T06:18:15.585 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.586 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.586 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.586 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "3", 2020-03-28T06:18:15.586 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.586 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.587 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "default replica count for any created local pool", 2020-03-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: "pg_num": { 2020-03-28T06:18:15.588 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "pg_num", 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "128", 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.589 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "default pg_num for any created local pool", 2020-03-28T06:18:15.590 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.590 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.590 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.590 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.590 INFO:teuthology.orchestra.run.smithi156.stdout: "prefix": { 2020-03-28T06:18:15.590 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "prefix", 2020-03-28T06:18:15.590 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "name prefix for any created local pool", 2020-03-28T06:18:15.591 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.592 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.592 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.592 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.592 INFO:teuthology.orchestra.run.smithi156.stdout: "subtree": { 2020-03-28T06:18:15.592 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "subtree", 2020-03-28T06:18:15.592 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.592 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.593 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.593 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "rack", 2020-03-28T06:18:15.593 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.593 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.593 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.593 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "CRUSH level for which to create a local pool", 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.594 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "orchestrator_cli", 2020-03-28T06:18:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: "orchestrator": { 2020-03-28T06:18:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "orchestrator", 2020-03-28T06:18:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.596 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.597 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.597 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.597 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.597 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.597 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.597 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "pg_autoscaler", 2020-03-28T06:18:15.597 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "sleep_interval": { 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "sleep_interval", 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.598 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "60", 2020-03-28T06:18:15.599 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.599 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.599 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.599 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.599 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.599 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.599 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.600 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.600 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.600 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.600 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.600 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "progress", 2020-03-28T06:18:15.600 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.600 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "max_completed_events": { 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "max_completed_events", 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "50", 2020-03-28T06:18:15.601 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "number of past completed events to remember", 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: "persist_interval": { 2020-03-28T06:18:15.603 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "persist_interval", 2020-03-28T06:18:15.603 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.603 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.603 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 1, 2020-03-28T06:18:15.603 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "5", 2020-03-28T06:18:15.603 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.603 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.604 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.604 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "how frequently to persist completed events", 2020-03-28T06:18:15.604 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.604 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.604 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.604 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.604 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "prometheus", 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: "rbd_stats_pools": { 2020-03-28T06:18:15.605 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rbd_stats_pools", 2020-03-28T06:18:15.606 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.606 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.606 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.606 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.606 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.606 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.606 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: "rbd_stats_pools_refresh_interval": { 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rbd_stats_pools_refresh_interval", 2020-03-28T06:18:15.607 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.608 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.608 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.608 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.608 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.608 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.608 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.608 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: "scrape_interval": { 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "scrape_interval", 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.609 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.610 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.610 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.610 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.610 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.610 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.610 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.610 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: "server_addr": { 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "server_addr", 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.611 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.612 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.612 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.612 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.612 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.612 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.612 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.612 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: "server_port": { 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "server_port", 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.613 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.614 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.614 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.614 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.614 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.614 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.614 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.614 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rbd_support", 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.615 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": {} 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "restful", 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: "key_file": { 2020-03-28T06:18:15.616 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "key_file", 2020-03-28T06:18:15.617 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.617 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.617 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.617 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.617 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.617 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.617 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.618 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.618 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.618 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.618 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.618 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.618 INFO:teuthology.orchestra.run.smithi156.stdout: "server_addr": { 2020-03-28T06:18:15.618 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "server_addr", 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.619 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.620 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.620 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.620 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.620 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.620 INFO:teuthology.orchestra.run.smithi156.stdout: "server_port": { 2020-03-28T06:18:15.620 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "server_port", 2020-03-28T06:18:15.620 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.621 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.621 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.621 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.621 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.621 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.621 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.622 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.622 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.622 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.622 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.622 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.622 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.622 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.623 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.623 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rook", 2020-03-28T06:18:15.623 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.623 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.623 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": {} 2020-03-28T06:18:15.623 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.623 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.624 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "selftest", 2020-03-28T06:18:15.624 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.624 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.624 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.624 INFO:teuthology.orchestra.run.smithi156.stdout: "roption1": { 2020-03-28T06:18:15.624 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "roption1", 2020-03-28T06:18:15.624 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.625 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.626 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.626 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.626 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.626 INFO:teuthology.orchestra.run.smithi156.stdout: "roption2": { 2020-03-28T06:18:15.626 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "roption2", 2020-03-28T06:18:15.626 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.626 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "xyz", 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.628 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.628 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.628 INFO:teuthology.orchestra.run.smithi156.stdout: "rwoption1": { 2020-03-28T06:18:15.628 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rwoption1", 2020-03-28T06:18:15.628 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.628 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.628 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.629 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.629 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.629 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.629 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.629 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.629 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.629 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: "rwoption2": { 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rwoption2", 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.630 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.631 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.631 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.631 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.631 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.631 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.631 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.631 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: "rwoption3": { 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rwoption3", 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "float", 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.632 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.633 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.633 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.633 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.633 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.633 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.633 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.633 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "rwoption4": { 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rwoption4", 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.634 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.635 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.635 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.635 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.635 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.635 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.635 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.635 INFO:teuthology.orchestra.run.smithi156.stdout: "rwoption5": { 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rwoption5", 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.636 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.637 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.637 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.637 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.637 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.637 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.637 INFO:teuthology.orchestra.run.smithi156.stdout: "rwoption6": { 2020-03-28T06:18:15.637 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "rwoption6", 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.638 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.639 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.639 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.639 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.639 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.639 INFO:teuthology.orchestra.run.smithi156.stdout: "testkey": { 2020-03-28T06:18:15.639 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "testkey", 2020-03-28T06:18:15.639 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.640 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.641 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.641 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.641 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.641 INFO:teuthology.orchestra.run.smithi156.stdout: "testlkey": { 2020-03-28T06:18:15.641 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "testlkey", 2020-03-28T06:18:15.641 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.641 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.642 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: "testnewline": { 2020-03-28T06:18:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "testnewline", 2020-03-28T06:18:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.645 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.645 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.645 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.645 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.645 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ssh", 2020-03-28T06:18:15.645 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.645 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "inventory_cache_timeout_min": { 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "inventory_cache_timeout_min", 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.646 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.647 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.647 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.647 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.647 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.647 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.647 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.647 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "ssh_config_file": { 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "ssh_config_file", 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.648 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.649 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.649 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.649 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.649 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.649 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.649 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.649 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "status", 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": {} 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.650 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.651 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "telegraf", 2020-03-28T06:18:15.651 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.651 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.651 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.651 INFO:teuthology.orchestra.run.smithi156.stdout: "address": { 2020-03-28T06:18:15.651 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "address", 2020-03-28T06:18:15.651 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.652 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.652 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.652 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2020-03-28T06:18:15.652 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.652 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.652 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.652 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: "interval": { 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "interval", 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.653 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.654 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.654 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "15", 2020-03-28T06:18:15.654 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.654 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.654 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.654 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.654 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "telemetry", 2020-03-28T06:18:15.655 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: "channel_basic": { 2020-03-28T06:18:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "channel_basic", 2020-03-28T06:18:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.656 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "Share basic cluster information (size, version)", 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.657 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.658 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.658 INFO:teuthology.orchestra.run.smithi156.stdout: "channel_crash": { 2020-03-28T06:18:15.658 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "channel_crash", 2020-03-28T06:18:15.658 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.658 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.658 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.658 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.659 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.660 INFO:teuthology.orchestra.run.smithi156.stdout: "channel_device": { 2020-03-28T06:18:15.660 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "channel_device", 2020-03-28T06:18:15.660 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.660 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.660 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.660 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "True", 2020-03-28T06:18:15.660 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout: "channel_ident": { 2020-03-28T06:18:15.662 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "channel_ident", 2020-03-28T06:18:15.662 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.662 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.662 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.662 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.662 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.662 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: "contact": { 2020-03-28T06:18:15.663 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "contact", 2020-03-28T06:18:15.664 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.664 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.664 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.664 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.664 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.664 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.664 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.665 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.665 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.665 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.665 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.665 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.665 INFO:teuthology.orchestra.run.smithi156.stdout: "description": { 2020-03-28T06:18:15.665 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "description", 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.666 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.667 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.667 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.667 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.667 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.667 INFO:teuthology.orchestra.run.smithi156.stdout: "device_url": { 2020-03-28T06:18:15.667 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "device_url", 2020-03-28T06:18:15.667 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "https://telemetry.ceph.com/device", 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.668 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.669 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.669 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.669 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.669 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": { 2020-03-28T06:18:15.669 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "enabled", 2020-03-28T06:18:15.669 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.669 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.670 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: "interval": { 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "interval", 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.671 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "24", 2020-03-28T06:18:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "8", 2020-03-28T06:18:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.672 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.673 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.673 INFO:teuthology.orchestra.run.smithi156.stdout: "last_opt_revision": { 2020-03-28T06:18:15.673 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "last_opt_revision", 2020-03-28T06:18:15.673 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.673 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.673 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.673 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "1", 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.674 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.675 INFO:teuthology.orchestra.run.smithi156.stdout: "leaderboard": { 2020-03-28T06:18:15.675 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "leaderboard", 2020-03-28T06:18:15.675 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "bool", 2020-03-28T06:18:15.675 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.675 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.675 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "False", 2020-03-28T06:18:15.675 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.676 INFO:teuthology.orchestra.run.smithi156.stdout: "organization": { 2020-03-28T06:18:15.677 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "organization", 2020-03-28T06:18:15.677 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.677 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.677 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.677 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.677 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.677 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: "proxy": { 2020-03-28T06:18:15.678 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "proxy", 2020-03-28T06:18:15.679 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.679 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.679 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.679 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.679 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.679 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.679 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.680 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.680 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.680 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.680 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.680 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.680 INFO:teuthology.orchestra.run.smithi156.stdout: "url": { 2020-03-28T06:18:15.680 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "url", 2020-03-28T06:18:15.681 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.681 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.681 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.681 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "https://telemetry.ceph.com/report", 2020-03-28T06:18:15.681 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.681 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.682 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "test_orchestrator", 2020-03-28T06:18:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": {} 2020-03-28T06:18:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.683 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.684 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "volumes", 2020-03-28T06:18:15.684 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.684 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.684 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": {} 2020-03-28T06:18:15.684 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.685 INFO:teuthology.orchestra.run.smithi156.stdout: { 2020-03-28T06:18:15.685 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "zabbix", 2020-03-28T06:18:15.685 INFO:teuthology.orchestra.run.smithi156.stdout: "can_run": true, 2020-03-28T06:18:15.685 INFO:teuthology.orchestra.run.smithi156.stdout: "error_string": "", 2020-03-28T06:18:15.685 INFO:teuthology.orchestra.run.smithi156.stdout: "module_options": { 2020-03-28T06:18:15.685 INFO:teuthology.orchestra.run.smithi156.stdout: "identifier": { 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "identifier", 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "", 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.686 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: "interval": { 2020-03-28T06:18:15.687 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "interval", 2020-03-28T06:18:15.688 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "secs", 2020-03-28T06:18:15.688 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.688 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.688 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "60", 2020-03-28T06:18:15.688 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.688 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.688 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.689 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.689 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.689 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.689 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.689 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.689 INFO:teuthology.orchestra.run.smithi156.stdout: "zabbix_host": { 2020-03-28T06:18:15.689 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "zabbix_host", 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "None", 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.690 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.691 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.691 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.691 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.691 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.691 INFO:teuthology.orchestra.run.smithi156.stdout: "zabbix_port": { 2020-03-28T06:18:15.691 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "zabbix_port", 2020-03-28T06:18:15.691 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "int", 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "10051", 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.692 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.693 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.693 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.693 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.693 INFO:teuthology.orchestra.run.smithi156.stdout: "zabbix_sender": { 2020-03-28T06:18:15.693 INFO:teuthology.orchestra.run.smithi156.stdout: "name": "zabbix_sender", 2020-03-28T06:18:15.693 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "str", 2020-03-28T06:18:15.693 INFO:teuthology.orchestra.run.smithi156.stdout: "level": "advanced", 2020-03-28T06:18:15.694 INFO:teuthology.orchestra.run.smithi156.stdout: "flags": 0, 2020-03-28T06:18:15.694 INFO:teuthology.orchestra.run.smithi156.stdout: "default_value": "/usr/bin/zabbix_sender", 2020-03-28T06:18:15.694 INFO:teuthology.orchestra.run.smithi156.stdout: "min": "", 2020-03-28T06:18:15.694 INFO:teuthology.orchestra.run.smithi156.stdout: "max": "", 2020-03-28T06:18:15.694 INFO:teuthology.orchestra.run.smithi156.stdout: "enum_allowed": [], 2020-03-28T06:18:15.694 INFO:teuthology.orchestra.run.smithi156.stdout: "desc": "", 2020-03-28T06:18:15.694 INFO:teuthology.orchestra.run.smithi156.stdout: "long_desc": "", 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: "tags": [], 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: "see_also": [] 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: "services": {}, 2020-03-28T06:18:15.695 INFO:teuthology.orchestra.run.smithi156.stdout: "always_on_modules": { 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "nautilus": [ 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "balancer", 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "crash", 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "devicehealth", 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "orchestrator_cli", 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "progress", 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "rbd_support", 2020-03-28T06:18:15.696 INFO:teuthology.orchestra.run.smithi156.stdout: "status", 2020-03-28T06:18:15.697 INFO:teuthology.orchestra.run.smithi156.stdout: "volumes" 2020-03-28T06:18:15.697 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2020-03-28T06:18:15.697 INFO:teuthology.orchestra.run.smithi156.stdout: } 2020-03-28T06:18:15.697 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.697 INFO:teuthology.orchestra.run.smithi156.stdout: "servicemap": { 2020-03-28T06:18:15.697 INFO:teuthology.orchestra.run.smithi156.stdout: "epoch": 1, 2020-03-28T06:18:15.697 INFO:teuthology.orchestra.run.smithi156.stdout: "modified": "0.000000", 2020-03-28T06:18:15.698 INFO:teuthology.orchestra.run.smithi156.stdout: "services": {} 2020-03-28T06:18:15.698 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2020-03-28T06:18:15.698 INFO:teuthology.orchestra.run.smithi156.stdout: "progress_events": {} 2020-03-28T06:18:15.698 INFO:teuthology.orchestra.run.smithi156.stdout:} 2020-03-28T06:18:15.700 INFO:tasks.ceph.ceph_manager.c1:{u'progress_events': {}, u'election_epoch': 3, u'quorum_age': 39, u'mgrmap': {u'available': True, u'active_change': u'2020-03-28 06:17:40.245487', u'available_modules': [{u'module_options': {u'smtp_port': {u'default_value': u'465', u'long_desc': u'', u'name': u'smtp_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'SMTP port'}, u'smtp_destination': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_destination', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Email address to send alerts to'}, u'smtp_sender': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_sender', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'SMTP envelope sender'}, u'interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'How frequently to reexamine health status'}, u'smtp_host': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_host', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'SMTP server'}, u'smtp_ssl': {u'default_value': u'True', u'long_desc': u'', u'name': u'smtp_ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'Use SSL to connect to SMTP server'}, u'smtp_from_name': {u'default_value': u'Ceph', u'long_desc': u'', u'name': u'smtp_from_name', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Email From: name'}, u'smtp_user': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_user', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'User to authenticate as'}, u'smtp_password': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Password to authenticate with'}}, u'name': u'alerts', u'error_string': u'', u'can_run': True}, {u'module_options': {u'username': {u'default_value': u'', u'long_desc': u'', u'name': u'username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'password': {u'default_value': u'', u'long_desc': u'', u'name': u'password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_url': {u'default_value': u'', u'long_desc': u'', u'name': u'server_url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'verify_server': {u'default_value': u'', u'long_desc': u'', u'name': u'verify_server', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'ansible', u'error_string': u'', u'can_run': True}, {u'module_options': {u'min_score': {u'default_value': u'0', u'long_desc': u'', u'name': u'min_score', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'float', u'tags': [], u'desc': u'minimum score, below which no optimization is attempted'}, u'end_weekday': {u'default_value': u'7', u'long_desc': u'0 or 7 = Sunday, 1 = Monday, etc.', u'name': u'end_weekday', u'min': u'0', u'max': u'7', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'Restrict automatic balancing to days of the week earlier than this'}, u'pool_ids': {u'default_value': u'', u'long_desc': u'', u'name': u'pool_ids', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'pools which the automatic balancing will be limited to'}, u'sleep_interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to wake up and attempt optimization'}, u'begin_time': {u'default_value': u'0000', u'long_desc': u'This is a time of day in the format HHMM.', u'name': u'begin_time', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'beginning time of day to automatically balance'}, u'crush_compat_metrics': {u'default_value': u'pgs,objects,bytes', u'long_desc': u'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', u'name': u'crush_compat_metrics', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'metrics with which to calculate OSD utilization'}, u'begin_weekday': {u'default_value': u'0', u'long_desc': u'0 or 7 = Sunday, 1 = Monday, etc.', u'name': u'begin_weekday', u'min': u'0', u'max': u'7', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'Restrict automatic balancing to this day of the week or later'}, u'upmap_max_iterations': {u'default_value': u'10', u'long_desc': u'', u'name': u'upmap_max_iterations', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'maximum upmap optimization iterations'}, u'upmap_max_deviation': {u'default_value': u'5', u'long_desc': u'If the number of PGs are within this count then no optimization is attempted', u'name': u'upmap_max_deviation', u'min': u'1', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'deviation below which no optimization is attempted'}, u'end_time': {u'default_value': u'2400', u'long_desc': u'This is a time of day in the format HHMM.', u'name': u'end_time', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'ending time of day to automatically balance'}, u'crush_compat_step': {u'default_value': u'0.5', u'long_desc': u'.99 is very aggressive, .01 is less aggressive', u'name': u'crush_compat_step', u'min': u'0.001', u'max': u'0.999', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'float', u'tags': [], u'desc': u'aggressiveness of optimization'}, u'crush_compat_max_iterations': {u'default_value': u'25', u'long_desc': u'', u'name': u'crush_compat_max_iterations', u'min': u'1', u'max': u'250', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'maximum number of iterations to attempt optimization'}, u'active': {u'default_value': u'False', u'long_desc': u'', u'name': u'active', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'automatically balance PGs across cluster'}, u'mode': {u'default_value': u'none', u'long_desc': u'', u'name': u'mode', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [u'crush-compat', u'none', u'upmap'], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Balancer mode'}}, u'name': u'balancer', u'error_string': u'', u'can_run': True}, {u'module_options': {u'warn_recent_interval': {u'default_value': u'1209600', u'long_desc': u'', u'name': u'warn_recent_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'time interval in which to warn about recent crashes'}, u'retain_interval': {u'default_value': u'31536000', u'long_desc': u'', u'name': u'retain_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how long to retain crashes before pruning them'}}, u'name': u'crash', u'error_string': u'', u'can_run': True}, {u'module_options': {u'FEATURE_TOGGLE_cephfs': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_cephfs', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_rbd': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_rbd', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'AUDIT_API_ENABLED': {u'default_value': u'False', u'long_desc': u'', u'name': u'AUDIT_API_ENABLED', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'RGW_API_SECRET_KEY': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_SECRET_KEY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ENABLE_BROWSABLE_API': {u'default_value': u'True', u'long_desc': u'', u'name': u'ENABLE_BROWSABLE_API', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'crt_file': {u'default_value': u'', u'long_desc': u'', u'name': u'crt_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'PROMETHEUS_API_HOST': {u'default_value': u'', u'long_desc': u'', u'name': u'PROMETHEUS_API_HOST', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'REST_REQUESTS_TIMEOUT': {u'default_value': u'45', u'long_desc': u'', u'name': u'REST_REQUESTS_TIMEOUT', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'GRAFANA_API_USERNAME': {u'default_value': u'admin', u'long_desc': u'', u'name': u'GRAFANA_API_USERNAME', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'standby_behaviour': {u'default_value': u'redirect', u'long_desc': u'', u'name': u'standby_behaviour', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [u'error', u'redirect'], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'url_prefix': {u'default_value': u'', u'long_desc': u'', u'name': u'url_prefix', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'password': {u'default_value': u'', u'long_desc': u'', u'name': u'password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_ADMIN_RESOURCE': {u'default_value': u'admin', u'long_desc': u'', u'name': u'RGW_API_ADMIN_RESOURCE', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_PORT': {u'default_value': u'80', u'long_desc': u'', u'name': u'RGW_API_PORT', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'jwt_token_ttl': {u'default_value': u'28800', u'long_desc': u'', u'name': u'jwt_token_ttl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'GRAFANA_API_URL': {u'default_value': u'', u'long_desc': u'', u'name': u'GRAFANA_API_URL', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ALERTMANAGER_API_HOST': {u'default_value': u'', u'long_desc': u'', u'name': u'ALERTMANAGER_API_HOST', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_iscsi': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_iscsi', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'server_port': {u'default_value': u'8080', u'long_desc': u'', u'name': u'server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {u'default_value': u'', u'long_desc': u'', u'name': u'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_ACCESS_KEY': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_ACCESS_KEY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'AUDIT_API_LOG_PAYLOAD': {u'default_value': u'True', u'long_desc': u'', u'name': u'AUDIT_API_LOG_PAYLOAD', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'RGW_API_HOST': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_HOST', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_SCHEME': {u'default_value': u'http', u'long_desc': u'', u'name': u'RGW_API_SCHEME', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'username': {u'default_value': u'', u'long_desc': u'', u'name': u'username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_SSL_VERIFY': {u'default_value': u'True', u'long_desc': u'', u'name': u'RGW_API_SSL_VERIFY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'RGW_API_USER_ID': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_USER_ID', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'GRAFANA_UPDATE_DASHBOARDS': {u'default_value': u'False', u'long_desc': u'', u'name': u'GRAFANA_UPDATE_DASHBOARDS', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_rgw': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_rgw', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'GRAFANA_API_PASSWORD': {u'default_value': u'admin', u'long_desc': u'', u'name': u'GRAFANA_API_PASSWORD', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ssl': {u'default_value': u'True', u'long_desc': u'', u'name': u'ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'standby_error_status_code': {u'default_value': u'500', u'long_desc': u'', u'name': u'standby_error_status_code', u'min': u'400', u'max': u'599', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'ISCSI_API_SSL_VERIFICATION': {u'default_value': u'True', u'long_desc': u'', u'name': u'ISCSI_API_SSL_VERIFICATION', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'ssl_server_port': {u'default_value': u'8443', u'long_desc': u'', u'name': u'ssl_server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'server_addr': {u'default_value': u'::', u'long_desc': u'', u'name': u'server_addr', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_mirroring': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_mirroring', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'debug': {u'default_value': u'False', u'long_desc': u'', u'name': u'debug', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'Enable/disable debug options'}, u'key_file': {u'default_value': u'', u'long_desc': u'', u'name': u'key_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'GRAFANA_API_SSL_VERIFY': {u'default_value': u'True', u'long_desc': u'', u'name': u'GRAFANA_API_SSL_VERIFY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}}, u'name': u'dashboard', u'error_string': u'', u'can_run': True}, {u'module_options': {u'salt_api_username': {u'default_value': u'', u'long_desc': u'', u'name': u'salt_api_username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'salt_api_password': {u'default_value': u'', u'long_desc': u'', u'name': u'salt_api_password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'salt_api_eauth': {u'default_value': u'sharedsecret', u'long_desc': u'', u'name': u'salt_api_eauth', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'salt_api_url': {u'default_value': u'', u'long_desc': u'', u'name': u'salt_api_url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'deepsea', u'error_string': u'', u'can_run': True}, {u'module_options': {u'pool_name': {u'default_value': u'device_health_metrics', u'long_desc': u'', u'name': u'pool_name', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'name of pool in which to store device health metrics'}, u'self_heal': {u'default_value': u'True', u'long_desc': u'', u'name': u'self_heal', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'preemptively heal cluster around devices that may fail'}, u'sleep_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to wake up and check device health'}, u'scrape_frequency': {u'default_value': u'86400', u'long_desc': u'', u'name': u'scrape_frequency', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to scrape device health metrics'}, u'enable_monitoring': {u'default_value': u'False', u'long_desc': u'', u'name': u'enable_monitoring', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'monitor device health metrics'}, u'retention_period': {u'default_value': u'15552000', u'long_desc': u'', u'name': u'retention_period', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how long to retain device health metrics'}, u'mark_out_threshold': {u'default_value': u'2419200', u'long_desc': u'', u'name': u'mark_out_threshold', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'automatically mark OSD if it may fail before this long'}, u'warn_threshold': {u'default_value': u'7257600', u'long_desc': u'', u'name': u'warn_threshold', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'raise health warning if OSD may fail before this long'}}, u'name': u'devicehealth', u'error_string': u'', u'can_run': True}, {u'module_options': {u'diskprediction_cert_context': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_cert_context', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_default_authority': {u'default_value': u'localhost', u'long_desc': u'', u'name': u'diskprediction_default_authority', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_ssl_target_name_override': {u'default_value': u'localhost', u'long_desc': u'', u'name': u'diskprediction_ssl_target_name_override', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'sleep_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_retrieve_prediction_interval': {u'default_value': u'43200', u'long_desc': u'', u'name': u'diskprediction_retrieve_prediction_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_upload_smart_interval': {u'default_value': u'43200', u'long_desc': u'', u'name': u'diskprediction_upload_smart_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_user': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_user', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_upload_metrics_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'diskprediction_upload_metrics_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_password': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_port': {u'default_value': u'31400', u'long_desc': u'', u'name': u'diskprediction_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_server': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_server', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'diskprediction_cloud', u'error_string': u'', u'can_run': True}, {u'module_options': {u'predict_interval': {u'default_value': u'86400', u'long_desc': u'', u'name': u'predict_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'sleep_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'diskprediction_local', u'error_string': u'', u'can_run': True}, {u'module_options': {u'username': {u'default_value': u'None', u'long_desc': u'', u'name': u'username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'interval': {u'default_value': u'30', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'database': {u'default_value': u'ceph', u'long_desc': u'', u'name': u'database', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'hostname': {u'default_value': u'None', u'long_desc': u'', u'name': u'hostname', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'batch_size': {u'default_value': u'5000', u'long_desc': u'', u'name': u'batch_size', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ssl': {u'default_value': u'false', u'long_desc': u'', u'name': u'ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'verify_ssl': {u'default_value': u'true', u'long_desc': u'', u'name': u'verify_ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'threads': {u'default_value': u'5', u'long_desc': u'', u'name': u'threads', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'password': {u'default_value': u'None', u'long_desc': u'', u'name': u'password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'port': {u'default_value': u'8086', u'long_desc': u'', u'name': u'port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'influx', u'error_string': u'influxdb python module not found', u'can_run': False}, {u'module_options': {}, u'name': u'insights', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'iostat', u'error_string': u'', u'can_run': True}, {u'module_options': {u'failure_domain': {u'default_value': u'host', u'long_desc': u'', u'name': u'failure_domain', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'failure domain for any created local pool'}, u'subtree': {u'default_value': u'rack', u'long_desc': u'', u'name': u'subtree', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'CRUSH level for which to create a local pool'}, u'min_size': {u'default_value': u'', u'long_desc': u'', u'name': u'min_size', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'default min_size for any created local pool'}, u'prefix': {u'default_value': u'', u'long_desc': u'', u'name': u'prefix', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'name prefix for any created local pool'}, u'pg_num': {u'default_value': u'128', u'long_desc': u'', u'name': u'pg_num', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'default pg_num for any created local pool'}, u'num_rep': {u'default_value': u'3', u'long_desc': u'', u'name': u'num_rep', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'default replica count for any created local pool'}}, u'name': u'localpool', u'error_string': u'', u'can_run': True}, {u'module_options': {u'orchestrator': {u'default_value': u'', u'long_desc': u'', u'name': u'orchestrator', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'orchestrator_cli', u'error_string': u'', u'can_run': True}, {u'module_options': {u'sleep_interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'pg_autoscaler', u'error_string': u'', u'can_run': True}, {u'module_options': {u'persist_interval': {u'default_value': u'5', u'long_desc': u'', u'name': u'persist_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to persist completed events'}, u'max_completed_events': {u'default_value': u'50', u'long_desc': u'', u'name': u'max_completed_events', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'number of past completed events to remember'}}, u'name': u'progress', u'error_string': u'', u'can_run': True}, {u'module_options': {u'rbd_stats_pools_refresh_interval': {u'default_value': u'', u'long_desc': u'', u'name': u'rbd_stats_pools_refresh_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rbd_stats_pools': {u'default_value': u'', u'long_desc': u'', u'name': u'rbd_stats_pools', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_port': {u'default_value': u'', u'long_desc': u'', u'name': u'server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_addr': {u'default_value': u'', u'long_desc': u'', u'name': u'server_addr', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'scrape_interval': {u'default_value': u'', u'long_desc': u'', u'name': u'scrape_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'prometheus', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'rbd_support', u'error_string': u'', u'can_run': True}, {u'module_options': {u'key_file': {u'default_value': u'', u'long_desc': u'', u'name': u'key_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_port': {u'default_value': u'', u'long_desc': u'', u'name': u'server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_addr': {u'default_value': u'', u'long_desc': u'', u'name': u'server_addr', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'restful', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'rook', u'error_string': u'', u'can_run': True}, {u'module_options': {u'testkey': {u'default_value': u'', u'long_desc': u'', u'name': u'testkey', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rwoption6': {u'default_value': u'True', u'long_desc': u'', u'name': u'rwoption6', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'rwoption5': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption5', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'roption2': {u'default_value': u'xyz', u'long_desc': u'', u'name': u'roption2', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'roption1': {u'default_value': u'', u'long_desc': u'', u'name': u'roption1', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rwoption4': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption4', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rwoption3': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption3', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'float', u'tags': [], u'desc': u''}, u'rwoption2': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption2', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'rwoption1': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption1', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'testnewline': {u'default_value': u'', u'long_desc': u'', u'name': u'testnewline', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'testlkey': {u'default_value': u'', u'long_desc': u'', u'name': u'testlkey', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'selftest', u'error_string': u'', u'can_run': True}, {u'module_options': {u'inventory_cache_timeout_min': {u'default_value': u'', u'long_desc': u'', u'name': u'inventory_cache_timeout_min', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ssh_config_file': {u'default_value': u'', u'long_desc': u'', u'name': u'ssh_config_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'ssh', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'status', u'error_string': u'', u'can_run': True}, {u'module_options': {u'interval': {u'default_value': u'15', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u''}, u'address': {u'default_value': u'unixgram:///tmp/telegraf.sock', u'long_desc': u'', u'name': u'address', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'telegraf', u'error_string': u'', u'can_run': True}, {u'module_options': {u'channel_crash': {u'default_value': u'True', u'long_desc': u'', u'name': u'channel_crash', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'channel_basic': {u'default_value': u'True', u'long_desc': u'', u'name': u'channel_basic', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'Share basic cluster information (size, version)'}, u'description': {u'default_value': u'None', u'long_desc': u'', u'name': u'description', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'url': {u'default_value': u'https://telemetry.ceph.com/report', u'long_desc': u'', u'name': u'url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'enabled': {u'default_value': u'False', u'long_desc': u'', u'name': u'enabled', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'interval': {u'default_value': u'24', u'long_desc': u'', u'name': u'interval', u'min': u'8', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'channel_device': {u'default_value': u'True', u'long_desc': u'', u'name': u'channel_device', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'channel_ident': {u'default_value': u'False', u'long_desc': u'', u'name': u'channel_ident', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'device_url': {u'default_value': u'https://telemetry.ceph.com/device', u'long_desc': u'', u'name': u'device_url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'contact': {u'default_value': u'None', u'long_desc': u'', u'name': u'contact', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'leaderboard': {u'default_value': u'False', u'long_desc': u'', u'name': u'leaderboard', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'organization': {u'default_value': u'None', u'long_desc': u'', u'name': u'organization', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'last_opt_revision': {u'default_value': u'1', u'long_desc': u'', u'name': u'last_opt_revision', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'proxy': {u'default_value': u'None', u'long_desc': u'', u'name': u'proxy', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'telemetry', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'test_orchestrator', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'volumes', u'error_string': u'', u'can_run': True}, {u'module_options': {u'zabbix_host': {u'default_value': u'None', u'long_desc': u'', u'name': u'zabbix_host', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'identifier': {u'default_value': u'', u'long_desc': u'', u'name': u'identifier', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'zabbix_sender': {u'default_value': u'/usr/bin/zabbix_sender', u'long_desc': u'', u'name': u'zabbix_sender', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'zabbix_port': {u'default_value': u'10051', u'long_desc': u'', u'name': u'zabbix_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u''}}, u'name': u'zabbix', u'error_string': u'', u'can_run': True}], u'always_on_modules': {u'nautilus': [u'balancer', u'crash', u'devicehealth', u'orchestrator_cli', u'progress', u'rbd_support', u'status', u'volumes']}, u'modules': [u'iostat', u'restful'], u'active_name': u'x', u'epoch': 3, u'active_addr': u'172.21.15.156:6801/8059', u'standbys': [], u'active_addrs': {u'addrvec': [{u'nonce': 8059, u'type': u'v2', u'addr': u'172.21.15.156:6800'}, {u'nonce': 8059, u'type': u'v1', u'addr': u'172.21.15.156:6801'}]}, u'services': {}, u'active_gid': 4102}, u'monmap': {u'features': {u'optional': [], u'persistent': [u'kraken', u'luminous', u'mimic', u'osdmap-prune', u'nautilus']}, u'created': u'2020-03-28 06:17:12.323554', u'min_mon_release': 14, u'modified': u'2020-03-28 06:17:12.323554', u'epoch': 1, u'mons': [{u'public_addr': u'172.21.15.156:6789/0', u'addr': u'172.21.15.156:6789/0', u'name': u'a', u'rank': 0, u'public_addrs': {u'addrvec': [{u'nonce': 0, u'type': u'v2', u'addr': u'172.21.15.156:3300'}, {u'nonce': 0, u'type': u'v1', u'addr': u'172.21.15.156:6789'}]}}], u'min_mon_release_name': u'nautilus', u'fsid': u'ff1cba1e-a6a7-4868-8184-daa87358bc60'}, u'health': {u'status': u'HEALTH_OK', u'checks': {}}, u'pgmap': {u'bytes_total': 322122547200, u'num_pgs': 8, u'data_bytes': 0, u'bytes_used': 3221594112, u'num_objects': 0, u'num_pools': 1, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+clean'}], u'bytes_avail': 318900953088}, u'quorum_names': [u'a'], u'fsmap': {u'up:standby': 0, u'epoch': 1, u'by_rank': []}, u'osdmap': {u'osdmap': {u'num_up_osds': 3, u'epoch': 13, u'num_in_osds': 3, u'num_osds': 3, u'num_remapped_pgs': 0}}, u'servicemap': {u'services': {}, u'epoch': 1, u'modified': u'0.000000'}, u'fsid': u'ff1cba1e-a6a7-4868-8184-daa87358bc60', u'quorum': [0]} 2020-03-28T06:18:15.702 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-03-28T06:18:15.702 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-03-28T06:18:15.702 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 health 2020-03-28T06:18:16.596 INFO:teuthology.misc.health.smithi156.stdout:HEALTH_OK 2020-03-28T06:18:16.613 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2020-03-28T06:18:16.613 INFO:teuthology.run_tasks:Running task ceph... 2020-03-28T06:18:16.663 INFO:tasks.ceph:Creating ceph cluster c2... 2020-03-28T06:18:16.664 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'c2', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'global': {'osd heartbeat grace': 40}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}, 'osd': {'debug osd': 25, 'osd fast shutdown': False, 'debug journal': 20, 'debug ms': 1, 'debug filestore': 20}, 'client': {'rgw crypt require ssl': False, 'rgw md log max shards': 4, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw data log num shards': 4, 'debug rgw': 20, 'rgw sync log trim interval': 0, 'rgw curl low speed time': 300}}} 2020-03-28T06:18:16.664 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'wait-for-scrub': False, 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}, 'cluster': 'c1', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'conf': {'global': {'osd heartbeat grace': 40}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}, 'osd': {'debug osd': 25, 'osd fast shutdown': False, 'debug journal': 20, 'debug ms': 1, 'debug filestore': 20}, 'client': {'rgw crypt require ssl': False, 'rgw md log max shards': 4, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw data log num shards': 4, 'debug rgw': 20, 'rgw sync log trim interval': 0, 'rgw curl low speed time': 300}}}}, {'ceph': {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'wait-for-scrub': False, 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}, 'cluster': 'c2', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'conf': {'global': {'osd heartbeat grace': 40}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}, 'osd': {'debug osd': 25, 'osd fast shutdown': False, 'debug journal': 20, 'debug ms': 1, 'debug filestore': 20}, 'client': {'rgw crypt require ssl': False, 'rgw md log max shards': 4, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw data log num shards': 4, 'debug rgw': 20, 'rgw sync log trim interval': 0, 'rgw curl low speed time': 300}}}}, {'rgw': {'c2.client.1': {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c2.client.0': {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c1.client.0': {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c1.client.1': {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.2393', 'owner': 'scheduled_kchai@teuthology', 'targets': {'smithi156.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVeY4umzW6UfyAQyF0mX5UySUYTGZr8AD8EWx3EzUiYESaXmRAx3cQSAbHtKkDkRQbTxFtAeOPifOGCXC1jA9BZHFSd8MAs/jswb2FmBxT60nVo6DtjeqWqjb4+7jAeNxo1Xj5fh+bpJKUE+XxXCWCgmtqsMHaBXN1nixUZwHtRE39ZZjfKLsAu45i122hiC9Mv7FKIhNyMSf+QqRX6jypdhrnMe8zHN2EuuFmT4GU/S7gok/a/mCWKf9FBber1PEVgW9FVmLCD26odAm6EnnA9sdVUW3cPxCxi4cYAU7/Bp7pqlmxTTC5Yir2ROfxun64//YQbMvxw/Ta765joJ9b', 'smithi168.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKIS5Dl34X/BN+J6deeiBJbAans534cCmppZyE7VNs0qxn6GJKDt0ONQ39VC2Hu/4nXeZepysnZ2kXoNXxiERPx+Jg0CHqE8VpXRuGLpJaHfm1saOesRPQcjD1h77kMdoyPKrRG2wGtsSqtPss6hF22R5RzhCspU493aK7JRKCv/lclpkrjBrZcw+gp7Vpd1zRTj0cBB7y51kC9lYB1HjL/j9d1mulBDvqIDWwPTHw7lazcMXO7ATxHTN7v94rWJQvoJr3l6r14DqEwcIK+1b9yu+wXnE1tSpBx26S7AicxpHfVqtflDQQcgGi0/bW2/3ZKL7XsjwLaqTgKYHqvxhp'}, 'job_id': '4896252', 'tube': 'smithi', 'suite_branch': 'nautilus-kefu2-testing-2020-03-27-2234', 'priority': 90, 'first_in_suite': False, 'branch': 'nautilus-kefu2-testing-2020-03-27-2234', 'suite': 'rgw', 'email': 'tchaikov@gmail.com', 'archive_path': '/home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252', 'description': 'rgw/multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'name': 'kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'os_type': 'centos', 'overrides': {'ceph': {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'wait-for-scrub': False, 'conf': {'global': {'osd heartbeat grace': 40}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}, 'osd': {'debug osd': 25, 'osd fast shutdown': False, 'debug journal': 20, 'debug ms': 1, 'debug filestore': 20}, 'client': {'rgw crypt require ssl': False, 'rgw md log max shards': 4, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw data log num shards': 4, 'debug rgw': 20, 'rgw sync log trim interval': 0, 'rgw curl low speed time': 300}}, 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'notcmalloc', 'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'zonegroups': [{'is_default': True, 'is_master': True, 'zones': [{'is_default': True, 'is_master': True, 'endpoints': ['c1.client.0'], 'name': 'test-zone1'}, {'is_default': True, 'endpoints': ['c2.client.0'], 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}], 'endpoints': ['c1.client.0'], 'name': 'test-zonegroup'}], 'realm': {'is default': True, 'name': 'test-realm'}}, 'admin_socket': {'branch': 'nautilus-kefu2-testing-2020-03-27-2234'}, 'workunit': {'sha1': 'cd131a1eb0d13cbc9e432c4911cf5025d18aef1e', 'branch': 'nautilus-kefu2-testing-2020-03-27-2234'}}} 2020-03-28T06:18:16.665 INFO:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-03-28T06:18:16.668 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-03-28T06:18:16.776 INFO:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2020-03-28T06:18:16.779 INFO:teuthology.orchestra.run.smithi168:> sudo install -d -m0777 -- /var/run/ceph 2020-03-28T06:18:16.963 DEBUG:teuthology.orchestra.remote:smithi168:/scratch_devs is 72B 2020-03-28T06:18:17.034 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-28T06:18:17.034 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-03-28T06:18:17.091 INFO:teuthology.orchestra.run.smithi168.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2020-03-28T06:18:17.091 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:18:17.091 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 5h/5d Inode: 101993 Links: 1 2020-03-28T06:18:17.091 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:18:17.091 INFO:teuthology.orchestra.run.smithi168.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:18:17.092 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-03-28 06:16:08.419598862 +0000 2020-03-28T06:18:17.092 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-03-28 06:14:23.112274892 +0000 2020-03-28T06:18:17.092 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-03-28 06:14:23.112274892 +0000 2020-03-28T06:18:17.092 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-03-28T06:18:17.110 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-03-28T06:18:17.110 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-03-28T06:18:17.110 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes (512 B) copied, 0.00044919 s, 1.1 MB/s 2020-03-28T06:18:17.115 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-03-28T06:18:17.177 INFO:teuthology.orchestra.run.smithi168.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2020-03-28T06:18:17.177 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:18:17.177 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 5h/5d Inode: 107824 Links: 1 2020-03-28T06:18:17.177 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:18:17.178 INFO:teuthology.orchestra.run.smithi168.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:18:17.178 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-03-28 06:16:08.419598862 +0000 2020-03-28T06:18:17.178 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-03-28 06:14:23.574275214 +0000 2020-03-28T06:18:17.178 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-03-28 06:14:23.574275214 +0000 2020-03-28T06:18:17.178 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-03-28T06:18:17.194 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-03-28T06:18:17.194 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-03-28T06:18:17.194 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes (512 B) copied, 0.000310617 s, 1.6 MB/s 2020-03-28T06:18:17.197 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-03-28T06:18:17.246 INFO:teuthology.orchestra.run.smithi168.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2020-03-28T06:18:17.246 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:18:17.246 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 5h/5d Inode: 102046 Links: 1 2020-03-28T06:18:17.246 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:18:17.246 INFO:teuthology.orchestra.run.smithi168.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:18:17.247 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-03-28 06:16:08.419598862 +0000 2020-03-28T06:18:17.247 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-03-28 06:14:24.033275533 +0000 2020-03-28T06:18:17.247 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-03-28 06:14:24.033275533 +0000 2020-03-28T06:18:17.247 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-03-28T06:18:17.263 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-03-28T06:18:17.264 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-03-28T06:18:17.264 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes (512 B) copied, 0.000438438 s, 1.2 MB/s 2020-03-28T06:18:17.267 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-03-28T06:18:17.315 INFO:teuthology.orchestra.run.smithi168.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2020-03-28T06:18:17.315 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-03-28T06:18:17.315 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 5h/5d Inode: 99821 Links: 1 2020-03-28T06:18:17.315 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-03-28T06:18:17.316 INFO:teuthology.orchestra.run.smithi168.stdout:Context: system_u:object_r:device_t:s0 2020-03-28T06:18:17.316 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-03-28 06:16:08.419598862 +0000 2020-03-28T06:18:17.316 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-03-28 06:14:24.493275852 +0000 2020-03-28T06:18:17.316 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-03-28 06:14:24.493275852 +0000 2020-03-28T06:18:17.316 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-03-28T06:18:17.330 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-03-28T06:18:17.330 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-03-28T06:18:17.330 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes (512 B) copied, 0.000297715 s, 1.7 MB/s 2020-03-28T06:18:17.333 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-03-28T06:18:17.333 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-03-28T06:18:17.333 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-03-28T06:18:17.333 INFO:tasks.ceph:dev map: {'c2.osd.1': '/dev/vg_nvme/lv_1', 'c2.osd.0': '/dev/vg_nvme/lv_4', 'c2.osd.2': '/dev/vg_nvme/lv_2'} 2020-03-28T06:18:17.333 INFO:tasks.ceph:Generating config... 2020-03-28T06:18:17.334 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-03-28T06:18:17.335 INFO:tasks.ceph:[mon] debug mon = 20 2020-03-28T06:18:17.335 INFO:tasks.ceph:[mon] debug paxos = 20 2020-03-28T06:18:17.335 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-03-28T06:18:17.335 INFO:tasks.ceph:[mon] debug ms = 1 2020-03-28T06:18:17.335 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-03-28T06:18:17.335 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-03-28T06:18:17.335 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-03-28T06:18:17.336 INFO:tasks.ceph:[osd] debug osd = 25 2020-03-28T06:18:17.336 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-03-28T06:18:17.336 INFO:tasks.ceph:[osd] debug journal = 20 2020-03-28T06:18:17.336 INFO:tasks.ceph:[osd] debug ms = 1 2020-03-28T06:18:17.336 INFO:tasks.ceph:[osd] debug filestore = 20 2020-03-28T06:18:17.336 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-03-28T06:18:17.336 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-03-28T06:18:17.336 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-03-28T06:18:17.337 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-03-28T06:18:17.337 INFO:tasks.ceph:[client] debug rgw = 20 2020-03-28T06:18:17.337 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-03-28T06:18:17.337 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-03-28T06:18:17.337 INFO:tasks.ceph:Setting up c2.mon.a... 2020-03-28T06:18:17.337 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-03-28T06:18:17.452 INFO:teuthology.orchestra.run.smithi168.stdout:creating /etc/ceph/c2.keyring 2020-03-28T06:18:17.455 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-03-28T06:18:17.592 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-03-28T06:18:17.679 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.168')] 2020-03-28T06:18:17.679 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.168 --print /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:17.799 INFO:teuthology.orchestra.run.smithi168.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:17.800 INFO:teuthology.orchestra.run.smithi168.stdout:monmaptool: generated fsid 2cc67b71-ad7c-4570-92b4-a0ade70c85c2 2020-03-28T06:18:17.800 INFO:teuthology.orchestra.run.smithi168.stdout:epoch 0 2020-03-28T06:18:17.800 INFO:teuthology.orchestra.run.smithi168.stdout:fsid 2cc67b71-ad7c-4570-92b4-a0ade70c85c2 2020-03-28T06:18:17.800 INFO:teuthology.orchestra.run.smithi168.stdout:last_changed 2020-03-28 06:18:17.802186 2020-03-28T06:18:17.800 INFO:teuthology.orchestra.run.smithi168.stdout:created 2020-03-28 06:18:17.802186 2020-03-28T06:18:17.801 INFO:teuthology.orchestra.run.smithi168.stdout:min_mon_release 0 (unknown) 2020-03-28T06:18:17.801 INFO:teuthology.orchestra.run.smithi168.stdout:0: [v2:172.21.15.168:3300/0,v1:172.21.15.168:6789/0] mon.a 2020-03-28T06:18:17.801 INFO:teuthology.orchestra.run.smithi168.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-03-28T06:18:17.802 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 2cc67b71-ad7c-4570-92b4-a0ade70c85c2... 2020-03-28T06:18:17.803 INFO:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf 2020-03-28T06:18:17.807 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf 2020-03-28T06:18:18.014 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-03-28T06:18:18.014 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-03-28T06:18:18.181 INFO:tasks.ceph:Copying monmap to all nodes... 2020-03-28T06:18:18.272 DEBUG:teuthology.orchestra.remote:smithi168:/etc/ceph/c2.keyring is 206B 2020-03-28T06:18:18.420 DEBUG:teuthology.orchestra.remote:smithi168:/home/ubuntu/cephtest/c2.monmap is 177B 2020-03-28T06:18:18.505 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2020-03-28T06:18:18.506 INFO:teuthology.orchestra.run.smithi156:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-03-28T06:18:18.636 INFO:teuthology.orchestra.run.smithi156:> cat > /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:18.703 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:18:18.703 INFO:teuthology.orchestra.run.smithi168:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-03-28T06:18:18.845 INFO:teuthology.orchestra.run.smithi168:> cat > /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:18.909 INFO:tasks.ceph:Setting up mon nodes... 2020-03-28T06:18:18.909 INFO:tasks.ceph:Setting up mgr nodes... 2020-03-28T06:18:18.910 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-03-28T06:18:19.121 INFO:teuthology.orchestra.run.smithi168.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-03-28T06:18:19.128 INFO:tasks.ceph:Setting up mds nodes... 2020-03-28T06:18:19.129 INFO:tasks.ceph_client:Setting up client nodes... 2020-03-28T06:18:19.129 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-03-28T06:18:19.299 INFO:teuthology.orchestra.run.smithi168.stdout:creating /etc/ceph/c2.client.0.keyring 2020-03-28T06:18:19.329 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-03-28T06:18:19.445 INFO:teuthology.orchestra.run.smithi168.stdout:creating /etc/ceph/c2.client.1.keyring 2020-03-28T06:18:19.472 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-03-28T06:18:19.472 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'c1.osd.2': '/dev/vg_nvme/lv_2', 'c1.osd.0': '/dev/vg_nvme/lv_4', 'c1.osd.1': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi168.front.sepia.ceph.com'): {'c2.osd.1': '/dev/vg_nvme/lv_1', 'c2.osd.0': '/dev/vg_nvme/lv_4', 'c2.osd.2': '/dev/vg_nvme/lv_2'}} 2020-03-28T06:18:19.472 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-03-28T06:18:19.563 INFO:tasks.ceph:{'c2.osd.1': '/dev/vg_nvme/lv_1', 'c2.osd.0': '/dev/vg_nvme/lv_4', 'c2.osd.2': '/dev/vg_nvme/lv_2'} 2020-03-28T06:18:19.563 INFO:tasks.ceph:{} 2020-03-28T06:18:19.563 INFO:tasks.ceph:c2.osd.0 2020-03-28T06:18:19.564 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:18:19.564 INFO:teuthology.orchestra.run.smithi168:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-03-28T06:18:20.136 INFO:teuthology.orchestra.run.smithi168.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-03-28T06:18:20.136 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-28T06:18:20.137 INFO:teuthology.orchestra.run.smithi168.stdout: = crc=1 finobt=0, sparse=0 2020-03-28T06:18:20.137 INFO:teuthology.orchestra.run.smithi168.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-28T06:18:20.137 INFO:teuthology.orchestra.run.smithi168.stdout: = sunit=0 swidth=0 blks 2020-03-28T06:18:20.137 INFO:teuthology.orchestra.run.smithi168.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-03-28T06:18:20.137 INFO:teuthology.orchestra.run.smithi168.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-28T06:18:20.137 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-28T06:18:20.137 INFO:teuthology.orchestra.run.smithi168.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-28T06:18:20.142 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi168.front.sepia.ceph.com -o noatime 2020-03-28T06:18:20.142 INFO:teuthology.orchestra.run.smithi168:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/c2-0 2020-03-28T06:18:20.343 INFO:teuthology.orchestra.run.smithi168:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-03-28T06:18:20.471 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-03-28T06:18:20.582 INFO:tasks.ceph:{'c2.osd.1': '/dev/vg_nvme/lv_1', 'c2.osd.0': '/dev/vg_nvme/lv_4', 'c2.osd.2': '/dev/vg_nvme/lv_2'} 2020-03-28T06:18:20.582 INFO:tasks.ceph:{} 2020-03-28T06:18:20.582 INFO:tasks.ceph:c2.osd.1 2020-03-28T06:18:20.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:18:20.583 INFO:teuthology.orchestra.run.smithi168:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-03-28T06:18:21.189 INFO:teuthology.orchestra.run.smithi168.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-03-28T06:18:21.189 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-28T06:18:21.190 INFO:teuthology.orchestra.run.smithi168.stdout: = crc=1 finobt=0, sparse=0 2020-03-28T06:18:21.190 INFO:teuthology.orchestra.run.smithi168.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-28T06:18:21.190 INFO:teuthology.orchestra.run.smithi168.stdout: = sunit=0 swidth=0 blks 2020-03-28T06:18:21.190 INFO:teuthology.orchestra.run.smithi168.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-03-28T06:18:21.190 INFO:teuthology.orchestra.run.smithi168.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-28T06:18:21.190 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-28T06:18:21.190 INFO:teuthology.orchestra.run.smithi168.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-28T06:18:21.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi168.front.sepia.ceph.com -o noatime 2020-03-28T06:18:21.192 INFO:teuthology.orchestra.run.smithi168:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-1 2020-03-28T06:18:21.344 INFO:teuthology.orchestra.run.smithi168:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-03-28T06:18:21.456 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-03-28T06:18:21.559 INFO:tasks.ceph:{'c2.osd.1': '/dev/vg_nvme/lv_1', 'c2.osd.0': '/dev/vg_nvme/lv_4', 'c2.osd.2': '/dev/vg_nvme/lv_2'} 2020-03-28T06:18:21.559 INFO:tasks.ceph:{} 2020-03-28T06:18:21.559 INFO:tasks.ceph:c2.osd.2 2020-03-28T06:18:21.559 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi168.front.sepia.ceph.com 2020-03-28T06:18:21.559 INFO:teuthology.orchestra.run.smithi168:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-03-28T06:18:22.172 INFO:teuthology.orchestra.run.smithi168.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-03-28T06:18:22.173 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 attr=2, projid32bit=1 2020-03-28T06:18:22.173 INFO:teuthology.orchestra.run.smithi168.stdout: = crc=1 finobt=0, sparse=0 2020-03-28T06:18:22.173 INFO:teuthology.orchestra.run.smithi168.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-03-28T06:18:22.173 INFO:teuthology.orchestra.run.smithi168.stdout: = sunit=0 swidth=0 blks 2020-03-28T06:18:22.173 INFO:teuthology.orchestra.run.smithi168.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-03-28T06:18:22.173 INFO:teuthology.orchestra.run.smithi168.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-03-28T06:18:22.174 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-03-28T06:18:22.174 INFO:teuthology.orchestra.run.smithi168.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-03-28T06:18:22.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi168.front.sepia.ceph.com -o noatime 2020-03-28T06:18:22.175 INFO:teuthology.orchestra.run.smithi168:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-2 2020-03-28T06:18:22.352 INFO:teuthology.orchestra.run.smithi168:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-03-28T06:18:22.456 INFO:teuthology.orchestra.run.smithi168:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:22.637 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:22.638 7f8b659fb9c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-03-28T06:18:22.638 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:22.639 7f8b659fb9c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-03-28T06:18:22.638 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:22.639 7f8b659fb9c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-03-28T06:18:22.638 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:22.639 7f8b659fb9c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-03-28T06:18:22.639 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:22.639 7f8b659fb9c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-03-28T06:18:22.639 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:22.640 7f8b659fb9c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-03-28T06:18:24.687 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-03-28T06:18:24.820 INFO:teuthology.orchestra.run.smithi168:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:24.991 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:24.992 7f2a7a9339c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-03-28T06:18:24.991 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:24.992 7f2a7a9339c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-03-28T06:18:24.992 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:24.992 7f2a7a9339c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-03-28T06:18:24.992 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:24.992 7f2a7a9339c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-03-28T06:18:24.992 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:24.992 7f2a7a9339c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-03-28T06:18:24.992 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:24.994 7f2a7a9339c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-03-28T06:18:27.045 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-03-28T06:18:27.162 INFO:teuthology.orchestra.run.smithi168:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:27.319 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:27.320 7ff0983919c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-03-28T06:18:27.319 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:27.320 7ff0983919c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-03-28T06:18:27.319 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:27.320 7ff0983919c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-03-28T06:18:27.320 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:27.320 7ff0983919c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-03-28T06:18:27.320 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:27.320 7ff0983919c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-03-28T06:18:27.320 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:18:27.322 7ff0983919c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-03-28T06:18:29.367 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-03-28T06:18:29.484 INFO:tasks.ceph:Reading keys from all nodes... 2020-03-28T06:18:29.484 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:18:29.562 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.GetLDU2IYx 2020-03-28T06:18:29.563 INFO:teuthology.orchestra.run.smithi168:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.GetLDU2IYx 2020-03-28T06:18:29.672 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.GetLDU2IYx 2020-03-28T06:18:29.841 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.GetLDU2IYx is 56B 2020-03-28T06:18:29.902 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.GetLDU2IYx 2020-03-28T06:18:29.962 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:18:30.088 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.hiHNw5TA9C 2020-03-28T06:18:30.089 INFO:teuthology.orchestra.run.smithi168:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.hiHNw5TA9C 2020-03-28T06:18:30.206 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.hiHNw5TA9C 2020-03-28T06:18:30.406 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.hiHNw5TA9C is 56B 2020-03-28T06:18:30.495 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.hiHNw5TA9C 2020-03-28T06:18:30.566 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:18:30.630 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.h1U5oHq86n 2020-03-28T06:18:30.630 INFO:teuthology.orchestra.run.smithi168:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.h1U5oHq86n 2020-03-28T06:18:30.718 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.h1U5oHq86n 2020-03-28T06:18:30.880 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.h1U5oHq86n is 56B 2020-03-28T06:18:30.933 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.h1U5oHq86n 2020-03-28T06:18:30.982 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:18:31.105 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.ibPRuqyXno 2020-03-28T06:18:31.105 INFO:teuthology.orchestra.run.smithi168:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.ibPRuqyXno 2020-03-28T06:18:31.206 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.ibPRuqyXno 2020-03-28T06:18:31.355 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.ibPRuqyXno is 56B 2020-03-28T06:18:31.413 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.ibPRuqyXno 2020-03-28T06:18:31.575 DEBUG:teuthology.orchestra.remote:smithi168:/etc/ceph/c2.client.0.keyring is 59B 2020-03-28T06:18:31.703 DEBUG:teuthology.orchestra.remote:smithi168:/etc/ceph/c2.client.1.keyring is 59B 2020-03-28T06:18:31.765 INFO:tasks.ceph:Adding keys to all mons... 2020-03-28T06:18:31.765 INFO:teuthology.orchestra.run.smithi168:> sudo tee -a /etc/ceph/c2.keyring 2020-03-28T06:18:31.833 INFO:teuthology.orchestra.run.smithi168.stdout:[mgr.x] 2020-03-28T06:18:31.833 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQAr7H5e4PtoBxAAAeIilXdROjgxgH71MFxm7Q== 2020-03-28T06:18:31.834 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.0] 2020-03-28T06:18:31.834 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQAu7H5e4n0nJhAAMHkkwcTg528MsYBXG986yg== 2020-03-28T06:18:31.834 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.1] 2020-03-28T06:18:31.834 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQAw7H5ecsIzOxAApa4vpeVumaM3ldpbJ3babA== 2020-03-28T06:18:31.834 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.2] 2020-03-28T06:18:31.834 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQAz7H5eQ+UnExAAO0/4o/wzTRQJQ5jB+H+QQA== 2020-03-28T06:18:31.834 INFO:teuthology.orchestra.run.smithi168.stdout:[client.0] 2020-03-28T06:18:31.835 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQAr7H5emrMBEhAAl/ovG3UyXLz1pFUNR9VbBg== 2020-03-28T06:18:31.835 INFO:teuthology.orchestra.run.smithi168.stdout:[client.1] 2020-03-28T06:18:31.835 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQAr7H5eI5ixGhAAK72LdxlX/vZ7BquupjwU8A== 2020-03-28T06:18:31.836 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-03-28T06:18:32.050 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-28T06:18:32.247 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-28T06:18:32.434 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-03-28T06:18:32.628 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-03-28T06:18:32.829 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-03-28T06:18:33.021 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-03-28T06:18:33.021 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-03-28T06:18:33.133 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-03-28T06:18:33.334 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-03-28T06:18:33.418 INFO:teuthology.orchestra.run.smithi168:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-03-28T06:18:33.481 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-03-28T06:18:33.482 DEBUG:teuthology.misc:running c2.mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c2.mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2020-03-28T06:18:33.482 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-03-28T06:18:33.482 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster c2 -i a 2020-03-28T06:18:33.485 INFO:tasks.ceph.c2.mon.a:Started 2020-03-28T06:18:33.485 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-03-28T06:18:33.486 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-03-28T06:18:33.486 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster c2 -i x 2020-03-28T06:18:33.489 INFO:tasks.ceph.c2.mgr.x:Started 2020-03-28T06:18:33.489 INFO:tasks.ceph:Setting crush tunables to default 2020-03-28T06:18:33.489 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c2 osd crush tunables default 2020-03-28T06:18:38.939 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:00:04.965 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:18:44.548 INFO:teuthology.orchestra.run.smithi168.stderr:adjusted tunables profile to default 2020-03-28T06:18:44.563 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-03-28T06:18:44.564 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:18:44.611 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.9QUWwbezNq 2020-03-28T06:18:44.612 INFO:teuthology.orchestra.run.smithi168:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.9QUWwbezNq 2020-03-28T06:18:44.673 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.9QUWwbezNq 2020-03-28T06:18:44.780 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.9QUWwbezNq is 37B 2020-03-28T06:18:44.828 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.9QUWwbezNq 2020-03-28T06:18:44.870 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:18:44.965 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.Jrshj9aghM 2020-03-28T06:18:44.965 INFO:teuthology.orchestra.run.smithi168:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.Jrshj9aghM 2020-03-28T06:18:45.028 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.Jrshj9aghM 2020-03-28T06:18:45.134 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.Jrshj9aghM is 37B 2020-03-28T06:18:45.183 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.Jrshj9aghM 2020-03-28T06:18:45.234 INFO:teuthology.orchestra.run.smithi168:> mktemp 2020-03-28T06:18:45.340 INFO:teuthology.orchestra.run.smithi168.stdout:/tmp/tmp.h1IGrMxDAO 2020-03-28T06:18:45.376 INFO:teuthology.orchestra.run.smithi168:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.h1IGrMxDAO 2020-03-28T06:18:45.460 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0666 /tmp/tmp.h1IGrMxDAO 2020-03-28T06:18:45.580 DEBUG:teuthology.orchestra.remote:smithi168:/tmp/tmp.h1IGrMxDAO is 37B 2020-03-28T06:18:45.642 INFO:teuthology.orchestra.run.smithi168:> rm -fr /tmp/tmp.h1IGrMxDAO 2020-03-28T06:18:45.688 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c2 osd new 47398007-8382-4102-8f48-9f30302b0f06 0 2020-03-28T06:18:46.877 INFO:teuthology.orchestra.run.smithi168.stdout:0 2020-03-28T06:18:46.892 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c2 osd new 349fabc2-eec5-4c12-83f6-89fc4074134c 1 2020-03-28T06:18:47.936 INFO:teuthology.orchestra.run.smithi168.stdout:1 2020-03-28T06:18:47.960 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c2 osd new 24523659-5172-4d89-9f52-8f0f338c90ee 2 2020-03-28T06:18:48.833 INFO:teuthology.orchestra.run.smithi168.stdout:2 2020-03-28T06:18:48.854 DEBUG:teuthology.misc:running c2.osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c2.osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2020-03-28T06:18:48.855 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-03-28T06:18:48.855 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.osd.0.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c2 -i 0 2020-03-28T06:18:48.858 INFO:tasks.ceph.c2.osd.0:Started 2020-03-28T06:18:48.858 DEBUG:teuthology.misc:running c2.osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c2.osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2020-03-28T06:18:48.858 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-03-28T06:18:48.859 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.osd.1.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c2 -i 1 2020-03-28T06:18:48.861 INFO:tasks.ceph.c2.osd.1:Started 2020-03-28T06:18:48.861 DEBUG:teuthology.misc:running c2.osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c2.osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck'] 2020-03-28T06:18:48.862 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-03-28T06:18:48.862 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.osd.2.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c2 -i 2 2020-03-28T06:18:48.867 INFO:tasks.ceph.c2.osd.2:Started 2020-03-28T06:18:48.868 INFO:tasks.ceph:Waiting for OSDs to come up 2020-03-28T06:18:49.170 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-03-28T06:18:50.082 INFO:teuthology.misc.health.smithi168.stdout: 2020-03-28T06:18:50.083 INFO:teuthology.misc.health.smithi168.stdout:{"epoch":5,"fsid":"2cc67b71-ad7c-4570-92b4-a0ade70c85c2","created":"2020-03-28 06:18:41.620009","modified":"2020-03-28 06:18:48.735821","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"47398007-8382-4102-8f48-9f30302b0f06","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"349fabc2-eec5-4c12-83f6-89fc4074134c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"24523659-5172-4d89-9f52-8f0f338c90ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:18:50.100 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-03-28T06:18:55.085 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:==00:00:00:06.086 9193== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:55.142 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:==00:00:00:06.142 9193== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:55.737 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:2020-03-28 06:18:55.724 9ae0e40 -1 Falling back to public interface 2020-03-28T06:18:55.931 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:==00:00:00:06.931 9193== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:55.942 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:==00:00:00:06.952 9192== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:56.029 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:==00:00:00:07.033 9192== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:56.103 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:==00:00:00:07.094 9194== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:56.157 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:==00:00:00:07.149 9194== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:56.332 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:==00:00:00:07.333 9193== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:56.341 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:==00:00:00:07.341 9193== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:56.401 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-03-28T06:18:56.592 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:2020-03-28 06:18:56.581 9ae0e40 -1 Falling back to public interface 2020-03-28T06:18:56.749 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:2020-03-28 06:18:56.738 9ae0e40 -1 Falling back to public interface 2020-03-28T06:18:56.790 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:==00:00:00:07.800 9192== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:56.935 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:==00:00:00:07.928 9194== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:57.151 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:==00:00:00:08.161 9192== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:57.169 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:==00:00:00:08.180 9192== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:57.279 INFO:teuthology.misc.health.smithi168.stdout: 2020-03-28T06:18:57.279 INFO:teuthology.misc.health.smithi168.stdout:{"epoch":5,"fsid":"2cc67b71-ad7c-4570-92b4-a0ade70c85c2","created":"2020-03-28 06:18:41.620009","modified":"2020-03-28 06:18:48.735821","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"47398007-8382-4102-8f48-9f30302b0f06","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"349fabc2-eec5-4c12-83f6-89fc4074134c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"24523659-5172-4d89-9f52-8f0f338c90ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:18:57.290 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-03-28T06:18:57.326 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:==00:00:00:08.318 9194== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:57.335 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:==00:00:00:08.327 9194== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:57.407 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:==00:00:00:08.408 9193== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:58.153 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:==00:00:00:09.163 9192== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:58.221 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:==00:00:00:09.213 9194== Warning: unimplemented fcntl command: 1038 2020-03-28T06:18:58.741 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:2020-03-28 06:18:58.725 9ae0e40 -1 osd.2 0 log_to_monitors {default=true} 2020-03-28T06:18:59.429 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:2020-03-28 06:18:59.427 9ae0e40 -1 osd.0 0 log_to_monitors {default=true} 2020-03-28T06:18:59.452 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:2020-03-28 06:18:59.450 9ae0e40 -1 osd.1 0 log_to_monitors {default=true} 2020-03-28T06:19:00.489 INFO:tasks.ceph.c2.osd.2.smithi168.stderr:2020-03-28 06:19:00.477 288b7700 -1 osd.2 0 waiting for initial osdmap 2020-03-28T06:19:01.605 INFO:tasks.ceph.c2.osd.0.smithi168.stderr:2020-03-28 06:19:01.598 288b7700 -1 osd.0 0 waiting for initial osdmap 2020-03-28T06:19:01.610 INFO:tasks.ceph.c2.osd.1.smithi168.stderr:2020-03-28 06:19:01.605 288b7700 -1 osd.1 0 waiting for initial osdmap 2020-03-28T06:19:03.594 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-03-28T06:19:04.452 INFO:teuthology.misc.health.smithi168.stdout: 2020-03-28T06:19:04.452 INFO:teuthology.misc.health.smithi168.stdout:{"epoch":9,"fsid":"2cc67b71-ad7c-4570-92b4-a0ade70c85c2","created":"2020-03-28 06:18:41.620009","modified":"2020-03-28 06:19:02.386774","last_up_change":"2020-03-28 06:19:02.386774","last_in_change":"2020-03-28 06:19:02.386774","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"47398007-8382-4102-8f48-9f30302b0f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":9192},{"type":"v1","addr":"172.21.15.168:6811","nonce":9192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6812","nonce":9192},{"type":"v1","addr":"172.21.15.168:6813","nonce":9192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6816","nonce":9192},{"type":"v1","addr":"172.21.15.168:6817","nonce":9192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":9192},{"type":"v1","addr":"172.21.15.168:6815","nonce":9192}]},"public_addr":"172.21.15.168:6811/9192","cluster_addr":"172.21.15.168:6813/9192","heartbeat_back_addr":"172.21.15.168:6817/9192","heartbeat_front_addr":"172.21.15.168:6815/9192","state":["exists","up"]},{"osd":1,"uuid":"349fabc2-eec5-4c12-83f6-89fc4074134c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6818","nonce":9194},{"type":"v1","addr":"172.21.15.168:6819","nonce":9194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":9194},{"type":"v1","addr":"172.21.15.168:6821","nonce":9194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6824","nonce":9194},{"type":"v1","addr":"172.21.15.168:6825","nonce":9194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":9194},{"type":"v1","addr":"172.21.15.168:6823","nonce":9194}]},"public_addr":"172.21.15.168:6819/9194","cluster_addr":"172.21.15.168:6821/9194","heartbeat_back_addr":"172.21.15.168:6825/9194","heartbeat_front_addr":"172.21.15.168:6823/9194","state":["exists","up"]},{"osd":2,"uuid":"24523659-5172-4d89-9f52-8f0f338c90ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":9193},{"type":"v1","addr":"172.21.15.168:6803","nonce":9193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6804","nonce":9193},{"type":"v1","addr":"172.21.15.168:6805","nonce":9193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":9193},{"type":"v1","addr":"172.21.15.168:6809","nonce":9193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":9193},{"type":"v1","addr":"172.21.15.168:6807","nonce":9193}]},"public_addr":"172.21.15.168:6803/9193","cluster_addr":"172.21.15.168:6805/9193","heartbeat_back_addr":"172.21.15.168:6809/9193","heartbeat_front_addr":"172.21.15.168:6807/9193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:19:04.471 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-03-28T06:19:04.471 INFO:tasks.ceph:Creating RBD pool 2020-03-28T06:19:04.472 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c2 osd pool create rbd 8 2020-03-28T06:19:06.010 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'rbd' created 2020-03-28T06:19:06.024 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-03-28T06:19:06.929 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rbd' on pool 'rbd' 2020-03-28T06:19:06.957 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-03-28T06:19:06.958 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 osd dump --format=json 2020-03-28T06:19:07.789 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:19:07.789 INFO:teuthology.orchestra.run.smithi168.stdout:{"epoch":11,"fsid":"2cc67b71-ad7c-4570-92b4-a0ade70c85c2","created":"2020-03-28 06:18:41.620009","modified":"2020-03-28 06:19:06.804410","last_up_change":"2020-03-28 06:19:02.386774","last_in_change":"2020-03-28 06:19:02.386774","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-03-28 06:19:05.220310","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"47398007-8382-4102-8f48-9f30302b0f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":9192},{"type":"v1","addr":"172.21.15.168:6811","nonce":9192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6812","nonce":9192},{"type":"v1","addr":"172.21.15.168:6813","nonce":9192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6816","nonce":9192},{"type":"v1","addr":"172.21.15.168:6817","nonce":9192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":9192},{"type":"v1","addr":"172.21.15.168:6815","nonce":9192}]},"public_addr":"172.21.15.168:6811/9192","cluster_addr":"172.21.15.168:6813/9192","heartbeat_back_addr":"172.21.15.168:6817/9192","heartbeat_front_addr":"172.21.15.168:6815/9192","state":["exists","up"]},{"osd":1,"uuid":"349fabc2-eec5-4c12-83f6-89fc4074134c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6818","nonce":9194},{"type":"v1","addr":"172.21.15.168:6819","nonce":9194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":9194},{"type":"v1","addr":"172.21.15.168:6821","nonce":9194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6824","nonce":9194},{"type":"v1","addr":"172.21.15.168:6825","nonce":9194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":9194},{"type":"v1","addr":"172.21.15.168:6823","nonce":9194}]},"public_addr":"172.21.15.168:6819/9194","cluster_addr":"172.21.15.168:6821/9194","heartbeat_back_addr":"172.21.15.168:6825/9194","heartbeat_front_addr":"172.21.15.168:6823/9194","state":["exists","up"]},{"osd":2,"uuid":"24523659-5172-4d89-9f52-8f0f338c90ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":9193},{"type":"v1","addr":"172.21.15.168:6803","nonce":9193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6804","nonce":9193},{"type":"v1","addr":"172.21.15.168:6805","nonce":9193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":9193},{"type":"v1","addr":"172.21.15.168:6809","nonce":9193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":9193},{"type":"v1","addr":"172.21.15.168:6807","nonce":9193}]},"public_addr":"172.21.15.168:6803/9193","cluster_addr":"172.21.15.168:6805/9193","heartbeat_back_addr":"172.21.15.168:6809/9193","heartbeat_front_addr":"172.21.15.168:6807/9193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:19:07.806 INFO:tasks.ceph.ceph_manager.c2:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-03-28 06:19:05.220310', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'warn', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'11', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-03-28T06:19:07.807 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 osd pool get rbd pg_num 2020-03-28T06:19:08.619 INFO:teuthology.orchestra.run.smithi168.stdout:pg_num: 8 2020-03-28T06:19:08.633 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-03-28T06:19:08.634 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-03-28T06:19:08.634 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 mgr dump --format=json 2020-03-28T06:19:09.737 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:19:09.759 INFO:teuthology.orchestra.run.smithi168.stdout:{"epoch":3,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6800","nonce":8270},{"type":"v1","addr":"172.21.15.168:6801","nonce":8270}]},"active_addr":"172.21.15.168:6801/8270","active_change":"2020-03-28 06:18:45.089663","available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"]}} 2020-03-28T06:19:09.761 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-03-28T06:19:10.064 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-03-28T06:19:11.056 INFO:teuthology.misc.health.smithi168.stdout: 2020-03-28T06:19:11.056 INFO:teuthology.misc.health.smithi168.stdout:{"epoch":13,"fsid":"2cc67b71-ad7c-4570-92b4-a0ade70c85c2","created":"2020-03-28 06:18:41.620009","modified":"2020-03-28 06:19:08.954712","last_up_change":"2020-03-28 06:19:02.386774","last_in_change":"2020-03-28 06:19:02.386774","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-03-28 06:19:05.220310","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"47398007-8382-4102-8f48-9f30302b0f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":9192},{"type":"v1","addr":"172.21.15.168:6811","nonce":9192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6812","nonce":9192},{"type":"v1","addr":"172.21.15.168:6813","nonce":9192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6816","nonce":9192},{"type":"v1","addr":"172.21.15.168:6817","nonce":9192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":9192},{"type":"v1","addr":"172.21.15.168:6815","nonce":9192}]},"public_addr":"172.21.15.168:6811/9192","cluster_addr":"172.21.15.168:6813/9192","heartbeat_back_addr":"172.21.15.168:6817/9192","heartbeat_front_addr":"172.21.15.168:6815/9192","state":["exists","up"]},{"osd":1,"uuid":"349fabc2-eec5-4c12-83f6-89fc4074134c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6818","nonce":9194},{"type":"v1","addr":"172.21.15.168:6819","nonce":9194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":9194},{"type":"v1","addr":"172.21.15.168:6821","nonce":9194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6824","nonce":9194},{"type":"v1","addr":"172.21.15.168:6825","nonce":9194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":9194},{"type":"v1","addr":"172.21.15.168:6823","nonce":9194}]},"public_addr":"172.21.15.168:6819/9194","cluster_addr":"172.21.15.168:6821/9194","heartbeat_back_addr":"172.21.15.168:6825/9194","heartbeat_front_addr":"172.21.15.168:6823/9194","state":["exists","up"]},{"osd":2,"uuid":"24523659-5172-4d89-9f52-8f0f338c90ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":9193},{"type":"v1","addr":"172.21.15.168:6803","nonce":9193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6804","nonce":9193},{"type":"v1","addr":"172.21.15.168:6805","nonce":9193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":9193},{"type":"v1","addr":"172.21.15.168:6809","nonce":9193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":9193},{"type":"v1","addr":"172.21.15.168:6807","nonce":9193}]},"public_addr":"172.21.15.168:6803/9193","cluster_addr":"172.21.15.168:6805/9193","heartbeat_back_addr":"172.21.15.168:6809/9193","heartbeat_front_addr":"172.21.15.168:6807/9193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:19:11.080 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-03-28T06:19:11.081 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 osd dump --format=json 2020-03-28T06:19:12.066 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:19:12.066 INFO:teuthology.orchestra.run.smithi168.stdout:{"epoch":13,"fsid":"2cc67b71-ad7c-4570-92b4-a0ade70c85c2","created":"2020-03-28 06:18:41.620009","modified":"2020-03-28 06:19:08.954712","last_up_change":"2020-03-28 06:19:02.386774","last_in_change":"2020-03-28 06:19:02.386774","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-03-28 06:19:05.220310","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"47398007-8382-4102-8f48-9f30302b0f06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":9192},{"type":"v1","addr":"172.21.15.168:6811","nonce":9192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6812","nonce":9192},{"type":"v1","addr":"172.21.15.168:6813","nonce":9192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6816","nonce":9192},{"type":"v1","addr":"172.21.15.168:6817","nonce":9192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":9192},{"type":"v1","addr":"172.21.15.168:6815","nonce":9192}]},"public_addr":"172.21.15.168:6811/9192","cluster_addr":"172.21.15.168:6813/9192","heartbeat_back_addr":"172.21.15.168:6817/9192","heartbeat_front_addr":"172.21.15.168:6815/9192","state":["exists","up"]},{"osd":1,"uuid":"349fabc2-eec5-4c12-83f6-89fc4074134c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6818","nonce":9194},{"type":"v1","addr":"172.21.15.168:6819","nonce":9194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":9194},{"type":"v1","addr":"172.21.15.168:6821","nonce":9194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6824","nonce":9194},{"type":"v1","addr":"172.21.15.168:6825","nonce":9194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":9194},{"type":"v1","addr":"172.21.15.168:6823","nonce":9194}]},"public_addr":"172.21.15.168:6819/9194","cluster_addr":"172.21.15.168:6821/9194","heartbeat_back_addr":"172.21.15.168:6825/9194","heartbeat_front_addr":"172.21.15.168:6823/9194","state":["exists","up"]},{"osd":2,"uuid":"24523659-5172-4d89-9f52-8f0f338c90ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":9193},{"type":"v1","addr":"172.21.15.168:6803","nonce":9193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6804","nonce":9193},{"type":"v1","addr":"172.21.15.168:6805","nonce":9193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":9193},{"type":"v1","addr":"172.21.15.168:6809","nonce":9193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":9193},{"type":"v1","addr":"172.21.15.168:6807","nonce":9193}]},"public_addr":"172.21.15.168:6803/9193","cluster_addr":"172.21.15.168:6805/9193","heartbeat_back_addr":"172.21.15.168:6809/9193","heartbeat_front_addr":"172.21.15.168:6807/9193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-03-28T06:19:12.087 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 tell osd.0 flush_pg_stats 2020-03-28T06:19:12.757 INFO:teuthology.orchestra.run.smithi168.stdout:38654705669 2020-03-28T06:19:12.775 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 tell osd.1 flush_pg_stats 2020-03-28T06:19:13.247 INFO:teuthology.orchestra.run.smithi168.stdout:38654705668 2020-03-28T06:19:13.258 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 tell osd.2 flush_pg_stats 2020-03-28T06:19:13.765 INFO:teuthology.orchestra.run.smithi168.stdout:34359738373 2020-03-28T06:19:13.785 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 osd last-stat-seq osd.0 2020-03-28T06:19:14.590 INFO:teuthology.orchestra.run.smithi168.stdout:38654705669 2020-03-28T06:19:14.604 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705669 got 38654705669 for osd.0 2020-03-28T06:19:14.605 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 osd last-stat-seq osd.1 2020-03-28T06:19:15.439 INFO:teuthology.orchestra.run.smithi168.stdout:38654705669 2020-03-28T06:19:15.453 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705669 for osd.1 2020-03-28T06:19:15.453 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 osd last-stat-seq osd.2 2020-03-28T06:19:16.395 INFO:teuthology.orchestra.run.smithi168.stdout:34359738373 2020-03-28T06:19:16.416 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738373 for osd.2 2020-03-28T06:19:16.417 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-03-28T06:19:16.417 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 pg dump --format=json 2020-03-28T06:19:17.370 INFO:teuthology.orchestra.run.smithi168.stderr:dumped all 2020-03-28T06:19:17.371 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:19:17.389 INFO:teuthology.orchestra.run.smithi168.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2020-03-28 06:19:17.238759","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":86736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.411198","last_change":"2020-03-28 06:19:08.368839","last_active":"2020-03-28 06:19:08.411198","last_peered":"2020-03-28 06:19:08.411198","last_clean":"2020-03-28 06:19:08.411198","last_became_active":"2020-03-28 06:19:08.347309","last_became_peered":"2020-03-28 06:19:08.347309","last_unstale":"2020-03-28 06:19:08.411198","last_undegraded":"2020-03-28 06:19:08.411198","last_fullsized":"2020-03-28 06:19:08.411198","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.407960","last_change":"2020-03-28 06:19:08.344791","last_active":"2020-03-28 06:19:08.407960","last_peered":"2020-03-28 06:19:08.407960","last_clean":"2020-03-28 06:19:08.407960","last_became_active":"2020-03-28 06:19:08.336407","last_became_peered":"2020-03-28 06:19:08.336407","last_unstale":"2020-03-28 06:19:08.407960","last_undegraded":"2020-03-28 06:19:08.407960","last_fullsized":"2020-03-28 06:19:08.407960","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.406839","last_change":"2020-03-28 06:19:08.386487","last_active":"2020-03-28 06:19:08.406839","last_peered":"2020-03-28 06:19:08.406839","last_clean":"2020-03-28 06:19:08.406839","last_became_active":"2020-03-28 06:19:08.331250","last_became_peered":"2020-03-28 06:19:08.331250","last_unstale":"2020-03-28 06:19:08.406839","last_undegraded":"2020-03-28 06:19:08.406839","last_fullsized":"2020-03-28 06:19:08.406839","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.408753","last_change":"2020-03-28 06:19:08.364843","last_active":"2020-03-28 06:19:08.408753","last_peered":"2020-03-28 06:19:08.408753","last_clean":"2020-03-28 06:19:08.408753","last_became_active":"2020-03-28 06:19:08.353624","last_became_peered":"2020-03-28 06:19:08.353624","last_unstale":"2020-03-28 06:19:08.408753","last_undegraded":"2020-03-28 06:19:08.408753","last_fullsized":"2020-03-28 06:19:08.408753","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.409411","last_change":"2020-03-28 06:19:08.395519","last_active":"2020-03-28 06:19:08.409411","last_peered":"2020-03-28 06:19:08.409411","last_clean":"2020-03-28 06:19:08.409411","last_became_active":"2020-03-28 06:19:08.384459","last_became_peered":"2020-03-28 06:19:08.384459","last_unstale":"2020-03-28 06:19:08.409411","last_undegraded":"2020-03-28 06:19:08.409411","last_fullsized":"2020-03-28 06:19:08.409411","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.404780","last_change":"2020-03-28 06:19:08.363499","last_active":"2020-03-28 06:19:08.404780","last_peered":"2020-03-28 06:19:08.404780","last_clean":"2020-03-28 06:19:08.404780","last_became_active":"2020-03-28 06:19:08.313930","last_became_peered":"2020-03-28 06:19:08.313930","last_unstale":"2020-03-28 06:19:08.404780","last_undegraded":"2020-03-28 06:19:08.404780","last_fullsized":"2020-03-28 06:19:08.404780","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.277047","last_change":"2020-03-28 06:19:07.437187","last_active":"2020-03-28 06:19:08.277047","last_peered":"2020-03-28 06:19:08.277047","last_clean":"2020-03-28 06:19:08.277047","last_became_active":"2020-03-28 06:19:07.387448","last_became_peered":"2020-03-28 06:19:07.387448","last_unstale":"2020-03-28 06:19:08.277047","last_undegraded":"2020-03-28 06:19:08.277047","last_fullsized":"2020-03-28 06:19:08.277047","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.405629","last_change":"2020-03-28 06:19:08.316501","last_active":"2020-03-28 06:19:08.405629","last_peered":"2020-03-28 06:19:08.405629","last_clean":"2020-03-28 06:19:08.405629","last_became_active":"2020-03-28 06:19:08.278638","last_became_peered":"2020-03-28 06:19:08.278638","last_unstale":"2020-03-28 06:19:08.405629","last_undegraded":"2020-03-28 06:19:08.405629","last_fullsized":"2020-03-28 06:19:08.405629","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-03-28T06:19:17.390 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 pg dump --format=json 2020-03-28T06:19:18.224 INFO:teuthology.orchestra.run.smithi168.stderr:dumped all 2020-03-28T06:19:18.224 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:19:18.241 INFO:teuthology.orchestra.run.smithi168.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2020-03-28 06:19:17.238759","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":86736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.411198","last_change":"2020-03-28 06:19:08.368839","last_active":"2020-03-28 06:19:08.411198","last_peered":"2020-03-28 06:19:08.411198","last_clean":"2020-03-28 06:19:08.411198","last_became_active":"2020-03-28 06:19:08.347309","last_became_peered":"2020-03-28 06:19:08.347309","last_unstale":"2020-03-28 06:19:08.411198","last_undegraded":"2020-03-28 06:19:08.411198","last_fullsized":"2020-03-28 06:19:08.411198","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.407960","last_change":"2020-03-28 06:19:08.344791","last_active":"2020-03-28 06:19:08.407960","last_peered":"2020-03-28 06:19:08.407960","last_clean":"2020-03-28 06:19:08.407960","last_became_active":"2020-03-28 06:19:08.336407","last_became_peered":"2020-03-28 06:19:08.336407","last_unstale":"2020-03-28 06:19:08.407960","last_undegraded":"2020-03-28 06:19:08.407960","last_fullsized":"2020-03-28 06:19:08.407960","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.406839","last_change":"2020-03-28 06:19:08.386487","last_active":"2020-03-28 06:19:08.406839","last_peered":"2020-03-28 06:19:08.406839","last_clean":"2020-03-28 06:19:08.406839","last_became_active":"2020-03-28 06:19:08.331250","last_became_peered":"2020-03-28 06:19:08.331250","last_unstale":"2020-03-28 06:19:08.406839","last_undegraded":"2020-03-28 06:19:08.406839","last_fullsized":"2020-03-28 06:19:08.406839","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.408753","last_change":"2020-03-28 06:19:08.364843","last_active":"2020-03-28 06:19:08.408753","last_peered":"2020-03-28 06:19:08.408753","last_clean":"2020-03-28 06:19:08.408753","last_became_active":"2020-03-28 06:19:08.353624","last_became_peered":"2020-03-28 06:19:08.353624","last_unstale":"2020-03-28 06:19:08.408753","last_undegraded":"2020-03-28 06:19:08.408753","last_fullsized":"2020-03-28 06:19:08.408753","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.409411","last_change":"2020-03-28 06:19:08.395519","last_active":"2020-03-28 06:19:08.409411","last_peered":"2020-03-28 06:19:08.409411","last_clean":"2020-03-28 06:19:08.409411","last_became_active":"2020-03-28 06:19:08.384459","last_became_peered":"2020-03-28 06:19:08.384459","last_unstale":"2020-03-28 06:19:08.409411","last_undegraded":"2020-03-28 06:19:08.409411","last_fullsized":"2020-03-28 06:19:08.409411","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.404780","last_change":"2020-03-28 06:19:08.363499","last_active":"2020-03-28 06:19:08.404780","last_peered":"2020-03-28 06:19:08.404780","last_clean":"2020-03-28 06:19:08.404780","last_became_active":"2020-03-28 06:19:08.313930","last_became_peered":"2020-03-28 06:19:08.313930","last_unstale":"2020-03-28 06:19:08.404780","last_undegraded":"2020-03-28 06:19:08.404780","last_fullsized":"2020-03-28 06:19:08.404780","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.277047","last_change":"2020-03-28 06:19:07.437187","last_active":"2020-03-28 06:19:08.277047","last_peered":"2020-03-28 06:19:08.277047","last_clean":"2020-03-28 06:19:08.277047","last_became_active":"2020-03-28 06:19:07.387448","last_became_peered":"2020-03-28 06:19:07.387448","last_unstale":"2020-03-28 06:19:08.277047","last_undegraded":"2020-03-28 06:19:08.277047","last_fullsized":"2020-03-28 06:19:08.277047","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"12","state":"active+clean","last_fresh":"2020-03-28 06:19:08.405629","last_change":"2020-03-28 06:19:08.316501","last_active":"2020-03-28 06:19:08.405629","last_peered":"2020-03-28 06:19:08.405629","last_clean":"2020-03-28 06:19:08.405629","last_became_active":"2020-03-28 06:19:08.278638","last_became_peered":"2020-03-28 06:19:08.278638","last_unstale":"2020-03-28 06:19:08.405629","last_undegraded":"2020-03-28 06:19:08.405629","last_fullsized":"2020-03-28 06:19:08.405629","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-03-28 06:19:05.661909","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-03-28 06:19:05.661909","last_clean_scrub_stamp":"2020-03-28 06:19:05.661909","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":28912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-03-28T06:19:18.242 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 status --format=json-pretty 2020-03-28T06:19:19.410 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-03-28T06:19:19.410 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-03-28T06:19:19.410 INFO:teuthology.orchestra.run.smithi168.stdout: "fsid": "2cc67b71-ad7c-4570-92b4-a0ade70c85c2", 2020-03-28T06:19:19.410 INFO:teuthology.orchestra.run.smithi168.stdout: "health": { 2020-03-28T06:19:19.410 INFO:teuthology.orchestra.run.smithi168.stdout: "checks": {}, 2020-03-28T06:19:19.411 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "HEALTH_OK" 2020-03-28T06:19:19.411 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.411 INFO:teuthology.orchestra.run.smithi168.stdout: "election_epoch": 3, 2020-03-28T06:19:19.411 INFO:teuthology.orchestra.run.smithi168.stdout: "quorum": [ 2020-03-28T06:19:19.411 INFO:teuthology.orchestra.run.smithi168.stdout: 0 2020-03-28T06:19:19.411 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.411 INFO:teuthology.orchestra.run.smithi168.stdout: "quorum_names": [ 2020-03-28T06:19:19.412 INFO:teuthology.orchestra.run.smithi168.stdout: "a" 2020-03-28T06:19:19.412 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.412 INFO:teuthology.orchestra.run.smithi168.stdout: "quorum_age": 37, 2020-03-28T06:19:19.412 INFO:teuthology.orchestra.run.smithi168.stdout: "monmap": { 2020-03-28T06:19:19.412 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 1, 2020-03-28T06:19:19.412 INFO:teuthology.orchestra.run.smithi168.stdout: "fsid": "2cc67b71-ad7c-4570-92b4-a0ade70c85c2", 2020-03-28T06:19:19.412 INFO:teuthology.orchestra.run.smithi168.stdout: "modified": "2020-03-28 06:18:17.802186", 2020-03-28T06:19:19.413 INFO:teuthology.orchestra.run.smithi168.stdout: "created": "2020-03-28 06:18:17.802186", 2020-03-28T06:19:19.413 INFO:teuthology.orchestra.run.smithi168.stdout: "min_mon_release": 14, 2020-03-28T06:19:19.413 INFO:teuthology.orchestra.run.smithi168.stdout: "min_mon_release_name": "nautilus", 2020-03-28T06:19:19.413 INFO:teuthology.orchestra.run.smithi168.stdout: "features": { 2020-03-28T06:19:19.413 INFO:teuthology.orchestra.run.smithi168.stdout: "persistent": [ 2020-03-28T06:19:19.413 INFO:teuthology.orchestra.run.smithi168.stdout: "kraken", 2020-03-28T06:19:19.413 INFO:teuthology.orchestra.run.smithi168.stdout: "luminous", 2020-03-28T06:19:19.414 INFO:teuthology.orchestra.run.smithi168.stdout: "mimic", 2020-03-28T06:19:19.414 INFO:teuthology.orchestra.run.smithi168.stdout: "osdmap-prune", 2020-03-28T06:19:19.415 INFO:teuthology.orchestra.run.smithi168.stdout: "nautilus" 2020-03-28T06:19:19.415 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.415 INFO:teuthology.orchestra.run.smithi168.stdout: "optional": [] 2020-03-28T06:19:19.415 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.415 INFO:teuthology.orchestra.run.smithi168.stdout: "mons": [ 2020-03-28T06:19:19.415 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.415 INFO:teuthology.orchestra.run.smithi168.stdout: "rank": 0, 2020-03-28T06:19:19.416 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "a", 2020-03-28T06:19:19.416 INFO:teuthology.orchestra.run.smithi168.stdout: "public_addrs": { 2020-03-28T06:19:19.416 INFO:teuthology.orchestra.run.smithi168.stdout: "addrvec": [ 2020-03-28T06:19:19.416 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.416 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "v2", 2020-03-28T06:19:19.416 INFO:teuthology.orchestra.run.smithi168.stdout: "addr": "172.21.15.168:3300", 2020-03-28T06:19:19.416 INFO:teuthology.orchestra.run.smithi168.stdout: "nonce": 0 2020-03-28T06:19:19.417 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.417 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.417 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "v1", 2020-03-28T06:19:19.417 INFO:teuthology.orchestra.run.smithi168.stdout: "addr": "172.21.15.168:6789", 2020-03-28T06:19:19.417 INFO:teuthology.orchestra.run.smithi168.stdout: "nonce": 0 2020-03-28T06:19:19.417 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.417 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: "addr": "172.21.15.168:6789/0", 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: "public_addr": "172.21.15.168:6789/0" 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: "osdmap": { 2020-03-28T06:19:19.418 INFO:teuthology.orchestra.run.smithi168.stdout: "osdmap": { 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 13, 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: "num_osds": 3, 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: "num_up_osds": 3, 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: "num_in_osds": 3, 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: "num_remapped_pgs": 0 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.419 INFO:teuthology.orchestra.run.smithi168.stdout: "pgmap": { 2020-03-28T06:19:19.420 INFO:teuthology.orchestra.run.smithi168.stdout: "pgs_by_state": [ 2020-03-28T06:19:19.420 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.420 INFO:teuthology.orchestra.run.smithi168.stdout: "state_name": "active+clean", 2020-03-28T06:19:19.420 INFO:teuthology.orchestra.run.smithi168.stdout: "count": 8 2020-03-28T06:19:19.420 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.420 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.420 INFO:teuthology.orchestra.run.smithi168.stdout: "num_pgs": 8, 2020-03-28T06:19:19.421 INFO:teuthology.orchestra.run.smithi168.stdout: "num_pools": 1, 2020-03-28T06:19:19.421 INFO:teuthology.orchestra.run.smithi168.stdout: "num_objects": 0, 2020-03-28T06:19:19.421 INFO:teuthology.orchestra.run.smithi168.stdout: "data_bytes": 0, 2020-03-28T06:19:19.421 INFO:teuthology.orchestra.run.smithi168.stdout: "bytes_used": 3221594112, 2020-03-28T06:19:19.421 INFO:teuthology.orchestra.run.smithi168.stdout: "bytes_avail": 318900953088, 2020-03-28T06:19:19.421 INFO:teuthology.orchestra.run.smithi168.stdout: "bytes_total": 322122547200 2020-03-28T06:19:19.421 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: "fsmap": { 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 1, 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: "by_rank": [], 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: "up:standby": 0 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: "mgrmap": { 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 3, 2020-03-28T06:19:19.422 INFO:teuthology.orchestra.run.smithi168.stdout: "active_gid": 4098, 2020-03-28T06:19:19.423 INFO:teuthology.orchestra.run.smithi168.stdout: "active_name": "x", 2020-03-28T06:19:19.423 INFO:teuthology.orchestra.run.smithi168.stdout: "active_addrs": { 2020-03-28T06:19:19.424 INFO:teuthology.orchestra.run.smithi168.stdout: "addrvec": [ 2020-03-28T06:19:19.424 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.424 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "v2", 2020-03-28T06:19:19.424 INFO:teuthology.orchestra.run.smithi168.stdout: "addr": "172.21.15.168:6800", 2020-03-28T06:19:19.424 INFO:teuthology.orchestra.run.smithi168.stdout: "nonce": 8270 2020-03-28T06:19:19.425 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.425 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.425 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "v1", 2020-03-28T06:19:19.425 INFO:teuthology.orchestra.run.smithi168.stdout: "addr": "172.21.15.168:6801", 2020-03-28T06:19:19.426 INFO:teuthology.orchestra.run.smithi168.stdout: "nonce": 8270 2020-03-28T06:19:19.426 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.426 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-03-28T06:19:19.426 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.426 INFO:teuthology.orchestra.run.smithi168.stdout: "active_addr": "172.21.15.168:6801/8270", 2020-03-28T06:19:19.427 INFO:teuthology.orchestra.run.smithi168.stdout: "active_change": "2020-03-28 06:18:45.089663", 2020-03-28T06:19:19.427 INFO:teuthology.orchestra.run.smithi168.stdout: "available": true, 2020-03-28T06:19:19.427 INFO:teuthology.orchestra.run.smithi168.stdout: "standbys": [], 2020-03-28T06:19:19.427 INFO:teuthology.orchestra.run.smithi168.stdout: "modules": [ 2020-03-28T06:19:19.427 INFO:teuthology.orchestra.run.smithi168.stdout: "iostat", 2020-03-28T06:19:19.427 INFO:teuthology.orchestra.run.smithi168.stdout: "restful" 2020-03-28T06:19:19.427 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.428 INFO:teuthology.orchestra.run.smithi168.stdout: "available_modules": [ 2020-03-28T06:19:19.428 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.429 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "alerts", 2020-03-28T06:19:19.429 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.429 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.429 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.429 INFO:teuthology.orchestra.run.smithi168.stdout: "interval": { 2020-03-28T06:19:19.429 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "interval", 2020-03-28T06:19:19.429 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.430 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.430 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.430 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "60", 2020-03-28T06:19:19.430 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.430 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.430 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.430 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "How frequently to reexamine health status", 2020-03-28T06:19:19.431 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.431 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.431 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.431 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.431 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_destination": { 2020-03-28T06:19:19.431 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_destination", 2020-03-28T06:19:19.431 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Email address to send alerts to", 2020-03-28T06:19:19.432 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.433 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.433 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.438 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_from_name": { 2020-03-28T06:19:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_from_name", 2020-03-28T06:19:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "Ceph", 2020-03-28T06:19:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Email From: name", 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_host": { 2020-03-28T06:19:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_host", 2020-03-28T06:19:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "SMTP server", 2020-03-28T06:19:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.443 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_password": { 2020-03-28T06:19:19.443 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_password", 2020-03-28T06:19:19.443 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.443 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.443 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.443 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Password to authenticate with", 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.444 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.445 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_port": { 2020-03-28T06:19:19.445 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_port", 2020-03-28T06:19:19.445 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.446 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.446 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.446 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "465", 2020-03-28T06:19:19.447 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.447 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.447 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.447 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "SMTP port", 2020-03-28T06:19:19.447 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.448 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.448 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.448 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.448 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_sender": { 2020-03-28T06:19:19.448 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_sender", 2020-03-28T06:19:19.448 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.448 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.449 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.449 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.449 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.449 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.449 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.449 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "SMTP envelope sender", 2020-03-28T06:19:19.450 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.450 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.450 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.450 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.450 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_ssl": { 2020-03-28T06:19:19.450 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_ssl", 2020-03-28T06:19:19.450 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.451 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.451 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.451 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.451 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.451 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.451 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.451 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Use SSL to connect to SMTP server", 2020-03-28T06:19:19.452 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.452 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.452 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.453 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.453 INFO:teuthology.orchestra.run.smithi168.stdout: "smtp_user": { 2020-03-28T06:19:19.453 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "smtp_user", 2020-03-28T06:19:19.453 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.453 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.453 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.453 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.454 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.454 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.454 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.454 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "User to authenticate as", 2020-03-28T06:19:19.454 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.454 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.454 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.455 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.455 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.455 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.455 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.455 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ansible", 2020-03-28T06:19:19.456 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.456 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.456 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.456 INFO:teuthology.orchestra.run.smithi168.stdout: "password": { 2020-03-28T06:19:19.456 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "password", 2020-03-28T06:19:19.456 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.456 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.457 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.457 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.457 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.457 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.457 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.457 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.457 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.458 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.458 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.458 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.458 INFO:teuthology.orchestra.run.smithi168.stdout: "server_url": { 2020-03-28T06:19:19.458 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "server_url", 2020-03-28T06:19:19.458 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.458 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.459 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.460 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.460 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.460 INFO:teuthology.orchestra.run.smithi168.stdout: "username": { 2020-03-28T06:19:19.460 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "username", 2020-03-28T06:19:19.460 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.460 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.460 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.461 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.461 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.461 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.461 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.461 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.461 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.461 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.462 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.462 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.462 INFO:teuthology.orchestra.run.smithi168.stdout: "verify_server": { 2020-03-28T06:19:19.462 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "verify_server", 2020-03-28T06:19:19.462 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.462 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.462 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.463 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.464 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.464 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.464 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.464 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.464 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "balancer", 2020-03-28T06:19:19.464 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.464 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.465 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.465 INFO:teuthology.orchestra.run.smithi168.stdout: "active": { 2020-03-28T06:19:19.465 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "active", 2020-03-28T06:19:19.465 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.465 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.465 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.465 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "automatically balance PGs across cluster", 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.466 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.467 INFO:teuthology.orchestra.run.smithi168.stdout: "begin_time": { 2020-03-28T06:19:19.467 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "begin_time", 2020-03-28T06:19:19.467 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.467 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.467 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.467 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "0000", 2020-03-28T06:19:19.467 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.468 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.468 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.468 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "beginning time of day to automatically balance", 2020-03-28T06:19:19.468 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "This is a time of day in the format HHMM.", 2020-03-28T06:19:19.468 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.468 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.468 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.469 INFO:teuthology.orchestra.run.smithi168.stdout: "begin_weekday": { 2020-03-28T06:19:19.469 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "begin_weekday", 2020-03-28T06:19:19.469 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "uint", 2020-03-28T06:19:19.469 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.469 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.469 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "0", 2020-03-28T06:19:19.469 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "0", 2020-03-28T06:19:19.470 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "7", 2020-03-28T06:19:19.470 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.471 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2020-03-28T06:19:19.471 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2020-03-28T06:19:19.471 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.471 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: "crush_compat_max_iterations": { 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "crush_compat_max_iterations", 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "uint", 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "25", 2020-03-28T06:19:19.472 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "1", 2020-03-28T06:19:19.473 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "250", 2020-03-28T06:19:19.473 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.473 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "maximum number of iterations to attempt optimization", 2020-03-28T06:19:19.473 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.473 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.473 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.473 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.474 INFO:teuthology.orchestra.run.smithi168.stdout: "crush_compat_metrics": { 2020-03-28T06:19:19.474 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "crush_compat_metrics", 2020-03-28T06:19:19.474 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.474 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.474 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.474 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "pgs,objects,bytes", 2020-03-28T06:19:19.475 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.475 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.475 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.475 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "metrics with which to calculate OSD utilization", 2020-03-28T06:19:19.475 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2020-03-28T06:19:19.475 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.475 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: "crush_compat_step": { 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "crush_compat_step", 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "float", 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "0.5", 2020-03-28T06:19:19.476 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "0.001", 2020-03-28T06:19:19.477 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "0.999", 2020-03-28T06:19:19.477 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.477 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "aggressiveness of optimization", 2020-03-28T06:19:19.477 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2020-03-28T06:19:19.477 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.477 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.478 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.478 INFO:teuthology.orchestra.run.smithi168.stdout: "end_time": { 2020-03-28T06:19:19.478 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "end_time", 2020-03-28T06:19:19.478 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.478 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.478 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "2400", 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "ending time of day to automatically balance", 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "This is a time of day in the format HHMM.", 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.479 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.480 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.480 INFO:teuthology.orchestra.run.smithi168.stdout: "end_weekday": { 2020-03-28T06:19:19.480 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "end_weekday", 2020-03-28T06:19:19.480 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "uint", 2020-03-28T06:19:19.480 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.480 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.481 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "7", 2020-03-28T06:19:19.481 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "0", 2020-03-28T06:19:19.481 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "7", 2020-03-28T06:19:19.481 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.481 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2020-03-28T06:19:19.481 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2020-03-28T06:19:19.481 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: "min_score": { 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "min_score", 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "float", 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.482 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "0", 2020-03-28T06:19:19.483 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.483 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.483 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.483 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "minimum score, below which no optimization is attempted", 2020-03-28T06:19:19.483 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.483 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.483 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.484 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.484 INFO:teuthology.orchestra.run.smithi168.stdout: "mode": { 2020-03-28T06:19:19.484 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "mode", 2020-03-28T06:19:19.484 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.484 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.484 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.484 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "none", 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [ 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: "crush-compat", 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: "none", 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: "upmap" 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.485 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Balancer mode", 2020-03-28T06:19:19.486 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.486 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.486 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.486 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.486 INFO:teuthology.orchestra.run.smithi168.stdout: "pool_ids": { 2020-03-28T06:19:19.486 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "pool_ids", 2020-03-28T06:19:19.486 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.487 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.487 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.487 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.487 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.487 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.487 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.488 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "pools which the automatic balancing will be limited to", 2020-03-28T06:19:19.488 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.488 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.488 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.488 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.488 INFO:teuthology.orchestra.run.smithi168.stdout: "sleep_interval": { 2020-03-28T06:19:19.488 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "sleep_interval", 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "60", 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.489 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "how frequently to wake up and attempt optimization", 2020-03-28T06:19:19.490 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.490 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.490 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.490 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.490 INFO:teuthology.orchestra.run.smithi168.stdout: "upmap_max_deviation": { 2020-03-28T06:19:19.490 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "upmap_max_deviation", 2020-03-28T06:19:19.490 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.491 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.491 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.491 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "5", 2020-03-28T06:19:19.491 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "1", 2020-03-28T06:19:19.491 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.491 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.491 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "deviation below which no optimization is attempted", 2020-03-28T06:19:19.492 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2020-03-28T06:19:19.492 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.492 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.492 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.492 INFO:teuthology.orchestra.run.smithi168.stdout: "upmap_max_iterations": { 2020-03-28T06:19:19.492 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "upmap_max_iterations", 2020-03-28T06:19:19.492 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "uint", 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "10", 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "maximum upmap optimization iterations", 2020-03-28T06:19:19.493 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.494 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.494 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.494 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.494 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.494 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.494 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.494 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "crash", 2020-03-28T06:19:19.495 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.495 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.495 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.495 INFO:teuthology.orchestra.run.smithi168.stdout: "retain_interval": { 2020-03-28T06:19:19.495 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "retain_interval", 2020-03-28T06:19:19.495 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.495 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "31536000", 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "how long to retain crashes before pruning them", 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.496 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.497 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.497 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.497 INFO:teuthology.orchestra.run.smithi168.stdout: "warn_recent_interval": { 2020-03-28T06:19:19.497 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "warn_recent_interval", 2020-03-28T06:19:19.497 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.497 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.497 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "1209600", 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "time interval in which to warn about recent crashes", 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.498 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.499 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.499 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.499 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.499 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.499 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "dashboard", 2020-03-28T06:19:19.499 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.499 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.500 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.500 INFO:teuthology.orchestra.run.smithi168.stdout: "ALERTMANAGER_API_HOST": { 2020-03-28T06:19:19.500 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ALERTMANAGER_API_HOST", 2020-03-28T06:19:19.500 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.500 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.500 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.500 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.501 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.502 INFO:teuthology.orchestra.run.smithi168.stdout: "AUDIT_API_ENABLED": { 2020-03-28T06:19:19.502 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "AUDIT_API_ENABLED", 2020-03-28T06:19:19.502 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.502 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.502 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.502 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.502 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.503 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.503 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.503 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.503 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.503 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.503 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.504 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.504 INFO:teuthology.orchestra.run.smithi168.stdout: "AUDIT_API_LOG_PAYLOAD": { 2020-03-28T06:19:19.504 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2020-03-28T06:19:19.504 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.504 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.504 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.504 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.505 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.505 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.505 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.505 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.505 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.505 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.505 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.506 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.506 INFO:teuthology.orchestra.run.smithi168.stdout: "ENABLE_BROWSABLE_API": { 2020-03-28T06:19:19.506 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ENABLE_BROWSABLE_API", 2020-03-28T06:19:19.506 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.506 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.507 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.507 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.507 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.507 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.507 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.507 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.508 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.508 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.508 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.508 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.508 INFO:teuthology.orchestra.run.smithi168.stdout: "FEATURE_TOGGLE_cephfs": { 2020-03-28T06:19:19.508 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "FEATURE_TOGGLE_cephfs", 2020-03-28T06:19:19.508 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.509 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.510 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.510 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.510 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.510 INFO:teuthology.orchestra.run.smithi168.stdout: "FEATURE_TOGGLE_iscsi": { 2020-03-28T06:19:19.510 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "FEATURE_TOGGLE_iscsi", 2020-03-28T06:19:19.510 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.510 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.511 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.512 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.512 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.512 INFO:teuthology.orchestra.run.smithi168.stdout: "FEATURE_TOGGLE_mirroring": { 2020-03-28T06:19:19.512 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "FEATURE_TOGGLE_mirroring", 2020-03-28T06:19:19.512 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.512 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.512 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.513 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.513 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.513 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.513 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.513 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.513 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.513 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.514 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.514 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.514 INFO:teuthology.orchestra.run.smithi168.stdout: "FEATURE_TOGGLE_rbd": { 2020-03-28T06:19:19.514 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "FEATURE_TOGGLE_rbd", 2020-03-28T06:19:19.514 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.514 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.515 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.515 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.515 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.515 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.515 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.516 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.516 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.516 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.516 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.516 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.516 INFO:teuthology.orchestra.run.smithi168.stdout: "FEATURE_TOGGLE_rgw": { 2020-03-28T06:19:19.516 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "FEATURE_TOGGLE_rgw", 2020-03-28T06:19:19.517 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.517 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.517 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.517 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.517 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.517 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.517 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2020-03-28T06:19:19.518 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.519 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.519 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.519 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.519 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.519 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.519 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.519 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.520 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.520 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.520 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.520 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.520 INFO:teuthology.orchestra.run.smithi168.stdout: "GRAFANA_API_PASSWORD": { 2020-03-28T06:19:19.520 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "GRAFANA_API_PASSWORD", 2020-03-28T06:19:19.520 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "admin", 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.521 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.522 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.522 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.522 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.522 INFO:teuthology.orchestra.run.smithi168.stdout: "GRAFANA_API_SSL_VERIFY": { 2020-03-28T06:19:19.522 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2020-03-28T06:19:19.522 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.522 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.523 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.524 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.524 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.524 INFO:teuthology.orchestra.run.smithi168.stdout: "GRAFANA_API_URL": { 2020-03-28T06:19:19.524 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "GRAFANA_API_URL", 2020-03-28T06:19:19.524 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.524 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.525 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.525 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.525 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.525 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.525 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.525 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: "GRAFANA_API_USERNAME": { 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "GRAFANA_API_USERNAME", 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.526 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.527 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.527 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "admin", 2020-03-28T06:19:19.527 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.527 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.527 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.527 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.527 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.528 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.528 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.528 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.528 INFO:teuthology.orchestra.run.smithi168.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2020-03-28T06:19:19.528 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2020-03-28T06:19:19.528 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.528 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.529 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.529 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.529 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.529 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.529 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.530 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.530 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.530 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.530 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.530 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.530 INFO:teuthology.orchestra.run.smithi168.stdout: "ISCSI_API_SSL_VERIFICATION": { 2020-03-28T06:19:19.531 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2020-03-28T06:19:19.531 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.531 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.531 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.531 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.531 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.531 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.532 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.532 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.532 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.532 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.532 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.532 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.532 INFO:teuthology.orchestra.run.smithi168.stdout: "PROMETHEUS_API_HOST": { 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "PROMETHEUS_API_HOST", 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.533 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: "REST_REQUESTS_TIMEOUT": { 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "REST_REQUESTS_TIMEOUT", 2020-03-28T06:19:19.534 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.535 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.535 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.535 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "45", 2020-03-28T06:19:19.535 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.535 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.536 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.536 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.536 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.536 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.536 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_ACCESS_KEY": { 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_ACCESS_KEY", 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.537 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.538 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.538 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.538 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.538 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.538 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.538 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.539 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.539 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_ADMIN_RESOURCE": { 2020-03-28T06:19:19.539 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2020-03-28T06:19:19.539 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.539 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.539 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.539 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "admin", 2020-03-28T06:19:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_HOST": { 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_HOST", 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_PORT": { 2020-03-28T06:19:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_PORT", 2020-03-28T06:19:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "80", 2020-03-28T06:19:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_SCHEME": { 2020-03-28T06:19:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_SCHEME", 2020-03-28T06:19:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "http", 2020-03-28T06:19:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_SECRET_KEY": { 2020-03-28T06:19:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_SECRET_KEY", 2020-03-28T06:19:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.548 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.548 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.548 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.548 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.548 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.548 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.548 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.549 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_SSL_VERIFY": { 2020-03-28T06:19:19.549 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_SSL_VERIFY", 2020-03-28T06:19:19.549 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.549 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.549 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.549 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.549 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.550 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.550 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.551 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.551 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.551 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.551 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.551 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.551 INFO:teuthology.orchestra.run.smithi168.stdout: "RGW_API_USER_ID": { 2020-03-28T06:19:19.551 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "RGW_API_USER_ID", 2020-03-28T06:19:19.552 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.552 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.552 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.552 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.552 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.552 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.552 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.553 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.553 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.553 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.553 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.553 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.553 INFO:teuthology.orchestra.run.smithi168.stdout: "crt_file": { 2020-03-28T06:19:19.553 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "crt_file", 2020-03-28T06:19:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.554 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: "debug": { 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "debug", 2020-03-28T06:19:19.555 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.556 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.556 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.556 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.556 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.556 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.556 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.556 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Enable/disable debug options", 2020-03-28T06:19:19.557 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.557 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.557 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.557 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.557 INFO:teuthology.orchestra.run.smithi168.stdout: "jwt_token_ttl": { 2020-03-28T06:19:19.557 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "jwt_token_ttl", 2020-03-28T06:19:19.557 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.558 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.558 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.558 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "28800", 2020-03-28T06:19:19.558 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.558 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.558 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.558 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: "key_file": { 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "key_file", 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.559 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.560 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.560 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.560 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.560 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.560 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.561 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.561 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.561 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.561 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.561 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.561 INFO:teuthology.orchestra.run.smithi168.stdout: "password": { 2020-03-28T06:19:19.561 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "password", 2020-03-28T06:19:19.562 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.562 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.562 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.562 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.562 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.562 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.562 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: "server_addr": { 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "server_addr", 2020-03-28T06:19:19.563 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.564 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.564 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.564 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "::", 2020-03-28T06:19:19.564 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.564 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.564 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.564 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.565 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.565 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.565 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.565 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.565 INFO:teuthology.orchestra.run.smithi168.stdout: "server_port": { 2020-03-28T06:19:19.565 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "server_port", 2020-03-28T06:19:19.565 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.566 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.566 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.566 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "8080", 2020-03-28T06:19:19.566 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.566 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.566 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.567 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.567 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.567 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.567 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.567 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.567 INFO:teuthology.orchestra.run.smithi168.stdout: "ssl": { 2020-03-28T06:19:19.567 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ssl", 2020-03-28T06:19:19.568 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.568 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.568 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.568 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.568 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.568 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.568 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.569 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.569 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.569 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.569 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.569 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.569 INFO:teuthology.orchestra.run.smithi168.stdout: "ssl_server_port": { 2020-03-28T06:19:19.569 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ssl_server_port", 2020-03-28T06:19:19.570 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.570 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.570 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.570 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "8443", 2020-03-28T06:19:19.570 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.570 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.570 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.571 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.571 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.571 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.571 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.571 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.572 INFO:teuthology.orchestra.run.smithi168.stdout: "standby_behaviour": { 2020-03-28T06:19:19.572 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "standby_behaviour", 2020-03-28T06:19:19.572 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.572 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.573 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.573 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "redirect", 2020-03-28T06:19:19.573 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.573 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.573 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [ 2020-03-28T06:19:19.573 INFO:teuthology.orchestra.run.smithi168.stdout: "error", 2020-03-28T06:19:19.573 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect" 2020-03-28T06:19:19.574 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.574 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.574 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.574 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.574 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.574 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.575 INFO:teuthology.orchestra.run.smithi168.stdout: "standby_error_status_code": { 2020-03-28T06:19:19.575 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "standby_error_status_code", 2020-03-28T06:19:19.575 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.575 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.575 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.575 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "500", 2020-03-28T06:19:19.575 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "400", 2020-03-28T06:19:19.576 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "599", 2020-03-28T06:19:19.576 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.576 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.576 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.576 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.576 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.576 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.577 INFO:teuthology.orchestra.run.smithi168.stdout: "url_prefix": { 2020-03-28T06:19:19.577 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "url_prefix", 2020-03-28T06:19:19.577 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.577 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.577 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.577 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.578 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.578 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.578 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.578 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.578 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.578 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.578 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.579 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.579 INFO:teuthology.orchestra.run.smithi168.stdout: "username": { 2020-03-28T06:19:19.579 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "username", 2020-03-28T06:19:19.579 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.579 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.579 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.579 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.580 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.580 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.580 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.580 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.580 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.580 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.580 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.581 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.581 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.581 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.581 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.581 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "deepsea", 2020-03-28T06:19:19.582 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.582 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.582 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.582 INFO:teuthology.orchestra.run.smithi168.stdout: "salt_api_eauth": { 2020-03-28T06:19:19.582 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "salt_api_eauth", 2020-03-28T06:19:19.582 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.582 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.583 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.583 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "sharedsecret", 2020-03-28T06:19:19.583 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.583 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.583 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.583 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.583 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.584 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.584 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.584 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.584 INFO:teuthology.orchestra.run.smithi168.stdout: "salt_api_password": { 2020-03-28T06:19:19.584 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "salt_api_password", 2020-03-28T06:19:19.584 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.585 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.585 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.585 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.585 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.585 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.585 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.585 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.586 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.586 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.586 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.586 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.586 INFO:teuthology.orchestra.run.smithi168.stdout: "salt_api_url": { 2020-03-28T06:19:19.586 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "salt_api_url", 2020-03-28T06:19:19.586 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.587 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.587 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.587 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.587 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.587 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.587 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.587 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.588 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.588 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.588 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.588 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.588 INFO:teuthology.orchestra.run.smithi168.stdout: "salt_api_username": { 2020-03-28T06:19:19.588 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "salt_api_username", 2020-03-28T06:19:19.588 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.589 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.589 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.589 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.589 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.589 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.589 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.589 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.590 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.590 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.590 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.590 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.590 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.590 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.590 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.591 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "devicehealth", 2020-03-28T06:19:19.591 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.591 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.591 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.591 INFO:teuthology.orchestra.run.smithi168.stdout: "enable_monitoring": { 2020-03-28T06:19:19.592 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "enable_monitoring", 2020-03-28T06:19:19.592 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.592 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.592 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.592 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.592 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.592 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.593 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.593 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "monitor device health metrics", 2020-03-28T06:19:19.593 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.593 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.593 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.593 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.593 INFO:teuthology.orchestra.run.smithi168.stdout: "mark_out_threshold": { 2020-03-28T06:19:19.594 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "mark_out_threshold", 2020-03-28T06:19:19.594 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.594 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.594 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.594 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "2419200", 2020-03-28T06:19:19.594 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.594 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "automatically mark OSD if it may fail before this long", 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: "pool_name": { 2020-03-28T06:19:19.595 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "pool_name", 2020-03-28T06:19:19.596 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.596 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.596 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.596 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "device_health_metrics", 2020-03-28T06:19:19.596 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.596 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.596 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.597 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "name of pool in which to store device health metrics", 2020-03-28T06:19:19.597 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.597 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.597 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.597 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.597 INFO:teuthology.orchestra.run.smithi168.stdout: "retention_period": { 2020-03-28T06:19:19.597 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "retention_period", 2020-03-28T06:19:19.598 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.598 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.598 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.598 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "15552000", 2020-03-28T06:19:19.598 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.598 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.599 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.599 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "how long to retain device health metrics", 2020-03-28T06:19:19.599 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.599 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.599 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.599 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.599 INFO:teuthology.orchestra.run.smithi168.stdout: "scrape_frequency": { 2020-03-28T06:19:19.600 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "scrape_frequency", 2020-03-28T06:19:19.600 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.600 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.600 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.600 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "86400", 2020-03-28T06:19:19.600 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.600 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.601 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.601 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "how frequently to scrape device health metrics", 2020-03-28T06:19:19.601 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.601 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.601 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.601 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.602 INFO:teuthology.orchestra.run.smithi168.stdout: "self_heal": { 2020-03-28T06:19:19.602 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "self_heal", 2020-03-28T06:19:19.602 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.602 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.602 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.602 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.603 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.603 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.603 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.604 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "preemptively heal cluster around devices that may fail", 2020-03-28T06:19:19.604 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.604 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.604 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.604 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.604 INFO:teuthology.orchestra.run.smithi168.stdout: "sleep_interval": { 2020-03-28T06:19:19.605 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "sleep_interval", 2020-03-28T06:19:19.605 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.605 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.605 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.605 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "600", 2020-03-28T06:19:19.605 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.605 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.606 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.606 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "how frequently to wake up and check device health", 2020-03-28T06:19:19.606 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.606 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.606 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.606 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.606 INFO:teuthology.orchestra.run.smithi168.stdout: "warn_threshold": { 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "warn_threshold", 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "7257600", 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.607 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.608 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "raise health warning if OSD may fail before this long", 2020-03-28T06:19:19.608 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.608 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.608 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.608 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.608 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.609 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.609 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.609 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_cloud", 2020-03-28T06:19:19.609 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.609 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.609 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.609 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_cert_context": { 2020-03-28T06:19:19.610 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_cert_context", 2020-03-28T06:19:19.610 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.610 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.610 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.610 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.610 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.610 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.611 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.611 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.611 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.611 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.611 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.611 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.611 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_default_authority": { 2020-03-28T06:19:19.612 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_default_authority", 2020-03-28T06:19:19.612 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.612 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.612 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.612 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "localhost", 2020-03-28T06:19:19.613 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.613 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.613 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.613 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.613 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.614 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.614 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.614 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.614 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_password": { 2020-03-28T06:19:19.614 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_password", 2020-03-28T06:19:19.614 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.614 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.615 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.615 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.615 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.615 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.615 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.615 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.615 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.616 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.616 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.616 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.616 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_port": { 2020-03-28T06:19:19.616 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_port", 2020-03-28T06:19:19.616 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.616 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.617 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.617 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "31400", 2020-03-28T06:19:19.617 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.617 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.617 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.617 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.617 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_retrieve_prediction_interval": { 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_retrieve_prediction_interval", 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.618 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.619 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "43200", 2020-03-28T06:19:19.619 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.619 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.619 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.619 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.619 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.619 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.620 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.620 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.620 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_server": { 2020-03-28T06:19:19.620 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_server", 2020-03-28T06:19:19.620 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.620 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.620 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.621 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.621 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.621 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.621 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.621 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.621 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.621 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.622 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.622 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.622 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_ssl_target_name_override": { 2020-03-28T06:19:19.622 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_ssl_target_name_override", 2020-03-28T06:19:19.622 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "localhost", 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.623 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.624 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.624 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.624 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.624 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_upload_metrics_interval": { 2020-03-28T06:19:19.624 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_upload_metrics_interval", 2020-03-28T06:19:19.624 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.624 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.625 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.625 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "600", 2020-03-28T06:19:19.625 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.625 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.625 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.625 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.625 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.626 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.626 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.626 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.626 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_upload_smart_interval": { 2020-03-28T06:19:19.626 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_upload_smart_interval", 2020-03-28T06:19:19.626 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.627 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.627 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.627 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "43200", 2020-03-28T06:19:19.627 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.627 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.627 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.628 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.628 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.628 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.628 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.628 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.628 INFO:teuthology.orchestra.run.smithi168.stdout: "diskprediction_user": { 2020-03-28T06:19:19.628 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_user", 2020-03-28T06:19:19.629 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.629 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.629 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.629 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.629 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.629 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.629 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.630 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.630 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.630 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.630 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.630 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.630 INFO:teuthology.orchestra.run.smithi168.stdout: "sleep_interval": { 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "sleep_interval", 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "600", 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.631 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.632 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.632 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.632 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.632 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.632 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.633 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.633 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.633 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.633 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "diskprediction_local", 2020-03-28T06:19:19.633 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.633 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.633 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.634 INFO:teuthology.orchestra.run.smithi168.stdout: "predict_interval": { 2020-03-28T06:19:19.634 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "predict_interval", 2020-03-28T06:19:19.634 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.634 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.634 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.634 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "86400", 2020-03-28T06:19:19.634 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.635 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.635 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.635 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.635 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.635 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.635 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.635 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.636 INFO:teuthology.orchestra.run.smithi168.stdout: "sleep_interval": { 2020-03-28T06:19:19.636 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "sleep_interval", 2020-03-28T06:19:19.636 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.636 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.636 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.636 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "600", 2020-03-28T06:19:19.636 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.637 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.637 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.637 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.637 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.637 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.637 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.638 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.638 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.638 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.638 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.638 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "influx", 2020-03-28T06:19:19.638 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": false, 2020-03-28T06:19:19.638 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "influxdb python module not found", 2020-03-28T06:19:19.639 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.639 INFO:teuthology.orchestra.run.smithi168.stdout: "batch_size": { 2020-03-28T06:19:19.639 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "batch_size", 2020-03-28T06:19:19.639 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.639 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.639 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.639 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "5000", 2020-03-28T06:19:19.640 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.640 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.640 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.640 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.640 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.640 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.641 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.641 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.641 INFO:teuthology.orchestra.run.smithi168.stdout: "database": { 2020-03-28T06:19:19.641 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "database", 2020-03-28T06:19:19.641 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.641 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.641 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.642 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "ceph", 2020-03-28T06:19:19.642 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.642 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.642 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.642 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.642 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.642 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.643 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.643 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.643 INFO:teuthology.orchestra.run.smithi168.stdout: "hostname": { 2020-03-28T06:19:19.643 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "hostname", 2020-03-28T06:19:19.643 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.644 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.644 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.644 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.644 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.644 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.644 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: "interval": { 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "interval", 2020-03-28T06:19:19.645 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.646 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.646 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.646 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "30", 2020-03-28T06:19:19.646 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.646 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.646 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.646 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.647 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.647 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.647 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.647 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.647 INFO:teuthology.orchestra.run.smithi168.stdout: "password": { 2020-03-28T06:19:19.647 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "password", 2020-03-28T06:19:19.647 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.648 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.648 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.648 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.648 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.648 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.648 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.648 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.649 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.649 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.649 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.649 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.649 INFO:teuthology.orchestra.run.smithi168.stdout: "port": { 2020-03-28T06:19:19.649 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "port", 2020-03-28T06:19:19.649 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.650 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.650 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.650 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "8086", 2020-03-28T06:19:19.650 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.650 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.650 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.650 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.651 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.651 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.651 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.651 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.651 INFO:teuthology.orchestra.run.smithi168.stdout: "ssl": { 2020-03-28T06:19:19.651 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ssl", 2020-03-28T06:19:19.651 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.652 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.652 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.652 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "false", 2020-03-28T06:19:19.652 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.652 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.652 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.652 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.653 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.653 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.653 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.653 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.653 INFO:teuthology.orchestra.run.smithi168.stdout: "threads": { 2020-03-28T06:19:19.653 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "threads", 2020-03-28T06:19:19.654 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.654 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.654 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.654 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "5", 2020-03-28T06:19:19.654 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.654 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.654 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.655 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.655 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.655 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.655 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.655 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.655 INFO:teuthology.orchestra.run.smithi168.stdout: "username": { 2020-03-28T06:19:19.655 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "username", 2020-03-28T06:19:19.656 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.656 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.656 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.656 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.656 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.656 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.656 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.657 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.657 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.657 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.657 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.657 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.657 INFO:teuthology.orchestra.run.smithi168.stdout: "verify_ssl": { 2020-03-28T06:19:19.657 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "verify_ssl", 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "true", 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.658 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.659 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.659 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.659 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.659 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.659 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.659 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.659 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.660 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "insights", 2020-03-28T06:19:19.660 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.660 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.660 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": {} 2020-03-28T06:19:19.660 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.660 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.660 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "iostat", 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": {} 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "localpool", 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.661 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.662 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.662 INFO:teuthology.orchestra.run.smithi168.stdout: "failure_domain": { 2020-03-28T06:19:19.662 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "failure_domain", 2020-03-28T06:19:19.662 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.662 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.662 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.662 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "host", 2020-03-28T06:19:19.663 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.663 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.663 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.663 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "failure domain for any created local pool", 2020-03-28T06:19:19.663 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.664 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.664 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.664 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.664 INFO:teuthology.orchestra.run.smithi168.stdout: "min_size": { 2020-03-28T06:19:19.664 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "min_size", 2020-03-28T06:19:19.664 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.664 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.665 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.665 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.665 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.665 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.665 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.665 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "default min_size for any created local pool", 2020-03-28T06:19:19.665 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.666 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.666 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.666 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.666 INFO:teuthology.orchestra.run.smithi168.stdout: "num_rep": { 2020-03-28T06:19:19.666 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "num_rep", 2020-03-28T06:19:19.666 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.666 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.667 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.667 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "3", 2020-03-28T06:19:19.667 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.667 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.667 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.667 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "default replica count for any created local pool", 2020-03-28T06:19:19.668 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.668 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.668 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.669 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.669 INFO:teuthology.orchestra.run.smithi168.stdout: "pg_num": { 2020-03-28T06:19:19.669 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "pg_num", 2020-03-28T06:19:19.669 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.669 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.669 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.669 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "128", 2020-03-28T06:19:19.670 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.670 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.670 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.670 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "default pg_num for any created local pool", 2020-03-28T06:19:19.670 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.670 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.670 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.671 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.671 INFO:teuthology.orchestra.run.smithi168.stdout: "prefix": { 2020-03-28T06:19:19.671 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "prefix", 2020-03-28T06:19:19.671 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.671 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.671 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.671 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.672 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.672 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.672 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.672 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "name prefix for any created local pool", 2020-03-28T06:19:19.672 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.672 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.673 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.674 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.675 INFO:teuthology.orchestra.run.smithi168.stdout: "subtree": { 2020-03-28T06:19:19.675 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "subtree", 2020-03-28T06:19:19.675 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.676 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.676 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.676 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "rack", 2020-03-28T06:19:19.676 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.676 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.676 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "CRUSH level for which to create a local pool", 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.677 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.678 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "orchestrator_cli", 2020-03-28T06:19:19.678 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.678 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.678 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.678 INFO:teuthology.orchestra.run.smithi168.stdout: "orchestrator": { 2020-03-28T06:19:19.678 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "orchestrator", 2020-03-28T06:19:19.678 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.679 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.679 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.679 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.679 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.679 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.679 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.679 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.680 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "pg_autoscaler", 2020-03-28T06:19:19.681 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.681 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.681 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.681 INFO:teuthology.orchestra.run.smithi168.stdout: "sleep_interval": { 2020-03-28T06:19:19.681 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "sleep_interval", 2020-03-28T06:19:19.681 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.681 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.682 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.682 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "60", 2020-03-28T06:19:19.682 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.682 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.682 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.682 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.682 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.683 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.683 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.683 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.683 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.683 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.683 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.683 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "progress", 2020-03-28T06:19:19.684 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.684 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.684 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.684 INFO:teuthology.orchestra.run.smithi168.stdout: "max_completed_events": { 2020-03-28T06:19:19.684 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "max_completed_events", 2020-03-28T06:19:19.684 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.684 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "50", 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "number of past completed events to remember", 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.685 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.686 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.686 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.686 INFO:teuthology.orchestra.run.smithi168.stdout: "persist_interval": { 2020-03-28T06:19:19.686 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "persist_interval", 2020-03-28T06:19:19.686 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.686 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.686 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 1, 2020-03-28T06:19:19.687 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "5", 2020-03-28T06:19:19.687 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.687 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.687 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.687 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "how frequently to persist completed events", 2020-03-28T06:19:19.687 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.687 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.688 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.688 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.688 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.688 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.688 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.688 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "prometheus", 2020-03-28T06:19:19.688 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.689 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.689 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.689 INFO:teuthology.orchestra.run.smithi168.stdout: "rbd_stats_pools": { 2020-03-28T06:19:19.689 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rbd_stats_pools", 2020-03-28T06:19:19.689 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.689 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.689 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.690 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.690 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.690 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.690 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.690 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.690 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.690 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: "rbd_stats_pools_refresh_interval": { 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rbd_stats_pools_refresh_interval", 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.691 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.692 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.692 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.692 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.692 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.692 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.692 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.692 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.693 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.693 INFO:teuthology.orchestra.run.smithi168.stdout: "scrape_interval": { 2020-03-28T06:19:19.693 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "scrape_interval", 2020-03-28T06:19:19.693 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.693 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.693 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.693 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.694 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.694 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.694 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.694 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.694 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.694 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.694 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.695 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.695 INFO:teuthology.orchestra.run.smithi168.stdout: "server_addr": { 2020-03-28T06:19:19.695 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "server_addr", 2020-03-28T06:19:19.695 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.695 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.695 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.695 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.696 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.696 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.696 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.696 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.696 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.696 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.696 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.697 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.697 INFO:teuthology.orchestra.run.smithi168.stdout: "server_port": { 2020-03-28T06:19:19.697 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "server_port", 2020-03-28T06:19:19.697 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.697 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.697 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.697 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.698 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.698 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.698 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.698 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.698 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.698 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.698 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.699 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.699 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.699 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.699 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.699 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rbd_support", 2020-03-28T06:19:19.699 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.699 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.700 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": {} 2020-03-28T06:19:19.700 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.700 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.700 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "restful", 2020-03-28T06:19:19.700 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.700 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.700 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "key_file": { 2020-03-28T06:19:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "key_file", 2020-03-28T06:19:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "server_addr": { 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "server_addr", 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "server_port": { 2020-03-28T06:19:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "server_port", 2020-03-28T06:19:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rook", 2020-03-28T06:19:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": {} 2020-03-28T06:19:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "selftest", 2020-03-28T06:19:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "roption1": { 2020-03-28T06:19:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "roption1", 2020-03-28T06:19:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "roption2": { 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "roption2", 2020-03-28T06:19:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "xyz", 2020-03-28T06:19:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "rwoption1": { 2020-03-28T06:19:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rwoption1", 2020-03-28T06:19:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: "rwoption2": { 2020-03-28T06:19:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rwoption2", 2020-03-28T06:19:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.715 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.715 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.715 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.715 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.715 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.715 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.715 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.716 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.716 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.716 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.716 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.716 INFO:teuthology.orchestra.run.smithi168.stdout: "rwoption3": { 2020-03-28T06:19:19.716 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rwoption3", 2020-03-28T06:19:19.716 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "float", 2020-03-28T06:19:19.717 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.717 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.717 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.717 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.717 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.717 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.717 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.718 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.718 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.718 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.718 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.718 INFO:teuthology.orchestra.run.smithi168.stdout: "rwoption4": { 2020-03-28T06:19:19.718 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rwoption4", 2020-03-28T06:19:19.718 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.719 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.719 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.719 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.719 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.719 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.719 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.719 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: "rwoption5": { 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rwoption5", 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.720 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.721 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.721 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.721 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.721 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.721 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.721 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.721 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.722 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.722 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.722 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.722 INFO:teuthology.orchestra.run.smithi168.stdout: "rwoption6": { 2020-03-28T06:19:19.722 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rwoption6", 2020-03-28T06:19:19.722 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.722 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.723 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.723 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.723 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.723 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.723 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.723 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.723 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.724 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.724 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.724 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.724 INFO:teuthology.orchestra.run.smithi168.stdout: "testkey": { 2020-03-28T06:19:19.724 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "testkey", 2020-03-28T06:19:19.724 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.724 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.725 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.726 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.726 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.726 INFO:teuthology.orchestra.run.smithi168.stdout: "testlkey": { 2020-03-28T06:19:19.726 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "testlkey", 2020-03-28T06:19:19.726 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.726 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.727 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.727 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.727 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.727 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.727 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.727 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: "testnewline": { 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "testnewline", 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.728 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.729 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.729 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.729 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.729 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.729 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.729 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.729 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ssh", 2020-03-28T06:19:19.730 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.731 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.731 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.731 INFO:teuthology.orchestra.run.smithi168.stdout: "inventory_cache_timeout_min": { 2020-03-28T06:19:19.731 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "inventory_cache_timeout_min", 2020-03-28T06:19:19.731 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.731 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.731 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.732 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.732 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.732 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.732 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.732 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.732 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.732 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: "ssh_config_file": { 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "ssh_config_file", 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.733 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.734 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.734 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.734 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.734 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.734 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.734 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.734 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.735 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.735 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.735 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.735 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.735 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "status", 2020-03-28T06:19:19.735 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.735 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": {} 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "telegraf", 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.736 INFO:teuthology.orchestra.run.smithi168.stdout: "address": { 2020-03-28T06:19:19.737 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "address", 2020-03-28T06:19:19.737 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.737 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.737 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.737 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2020-03-28T06:19:19.737 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.737 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: "interval": { 2020-03-28T06:19:19.738 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "interval", 2020-03-28T06:19:19.739 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.739 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.739 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.739 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "15", 2020-03-28T06:19:19.739 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.739 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.739 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.740 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.740 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.740 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.740 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.740 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.740 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.740 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.741 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.741 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "telemetry", 2020-03-28T06:19:19.741 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.741 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.741 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.741 INFO:teuthology.orchestra.run.smithi168.stdout: "channel_basic": { 2020-03-28T06:19:19.741 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "channel_basic", 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.742 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "Share basic cluster information (size, version)", 2020-03-28T06:19:19.743 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.743 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.743 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.743 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.743 INFO:teuthology.orchestra.run.smithi168.stdout: "channel_crash": { 2020-03-28T06:19:19.743 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "channel_crash", 2020-03-28T06:19:19.743 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.744 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.744 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.744 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.744 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.744 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.744 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.744 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.745 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.745 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.745 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.745 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.745 INFO:teuthology.orchestra.run.smithi168.stdout: "channel_device": { 2020-03-28T06:19:19.745 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "channel_device", 2020-03-28T06:19:19.745 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "True", 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.746 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.747 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.747 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.747 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.747 INFO:teuthology.orchestra.run.smithi168.stdout: "channel_ident": { 2020-03-28T06:19:19.747 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "channel_ident", 2020-03-28T06:19:19.747 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.747 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.748 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.748 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.748 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.748 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.748 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.748 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.748 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.749 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.749 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.749 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.749 INFO:teuthology.orchestra.run.smithi168.stdout: "contact": { 2020-03-28T06:19:19.749 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "contact", 2020-03-28T06:19:19.749 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.749 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.750 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.751 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.751 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.751 INFO:teuthology.orchestra.run.smithi168.stdout: "description": { 2020-03-28T06:19:19.751 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "description", 2020-03-28T06:19:19.751 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.751 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.751 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.752 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.752 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.752 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.752 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.752 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.752 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.752 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.753 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.753 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.753 INFO:teuthology.orchestra.run.smithi168.stdout: "device_url": { 2020-03-28T06:19:19.753 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "device_url", 2020-03-28T06:19:19.753 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.753 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.753 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "https://telemetry.ceph.com/device", 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.754 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.755 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.755 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": { 2020-03-28T06:19:19.755 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "enabled", 2020-03-28T06:19:19.755 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.755 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.755 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.755 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.756 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.756 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.756 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.756 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.756 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.756 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.756 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.757 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.757 INFO:teuthology.orchestra.run.smithi168.stdout: "interval": { 2020-03-28T06:19:19.757 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "interval", 2020-03-28T06:19:19.757 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.757 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.757 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.757 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "24", 2020-03-28T06:19:19.758 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "8", 2020-03-28T06:19:19.758 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.758 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.758 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.758 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.758 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.758 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: "last_opt_revision": { 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "last_opt_revision", 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "1", 2020-03-28T06:19:19.759 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.760 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.760 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.760 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.760 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.760 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.760 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: "leaderboard": { 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "leaderboard", 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "bool", 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "False", 2020-03-28T06:19:19.761 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.762 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.762 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.762 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.762 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.762 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.762 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.762 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.763 INFO:teuthology.orchestra.run.smithi168.stdout: "organization": { 2020-03-28T06:19:19.763 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "organization", 2020-03-28T06:19:19.763 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.763 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.763 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.763 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.763 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.764 INFO:teuthology.orchestra.run.smithi168.stdout: "proxy": { 2020-03-28T06:19:19.765 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "proxy", 2020-03-28T06:19:19.765 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.765 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.765 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.765 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.765 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.765 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.766 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.766 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.766 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.766 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.766 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.766 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.766 INFO:teuthology.orchestra.run.smithi168.stdout: "url": { 2020-03-28T06:19:19.767 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "url", 2020-03-28T06:19:19.767 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.767 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.767 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.767 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "https://telemetry.ceph.com/report", 2020-03-28T06:19:19.767 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.767 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.768 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.768 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.768 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.768 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.768 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.768 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.768 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test_orchestrator", 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": {} 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.769 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.770 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "volumes", 2020-03-28T06:19:19.770 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.770 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.770 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": {} 2020-03-28T06:19:19.770 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.770 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-03-28T06:19:19.770 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "zabbix", 2020-03-28T06:19:19.771 INFO:teuthology.orchestra.run.smithi168.stdout: "can_run": true, 2020-03-28T06:19:19.771 INFO:teuthology.orchestra.run.smithi168.stdout: "error_string": "", 2020-03-28T06:19:19.771 INFO:teuthology.orchestra.run.smithi168.stdout: "module_options": { 2020-03-28T06:19:19.771 INFO:teuthology.orchestra.run.smithi168.stdout: "identifier": { 2020-03-28T06:19:19.771 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "identifier", 2020-03-28T06:19:19.771 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.771 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "", 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.772 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.773 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.773 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.773 INFO:teuthology.orchestra.run.smithi168.stdout: "interval": { 2020-03-28T06:19:19.773 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "interval", 2020-03-28T06:19:19.773 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "secs", 2020-03-28T06:19:19.773 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.773 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.774 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "60", 2020-03-28T06:19:19.774 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.774 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.774 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.774 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.774 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.774 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.775 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.775 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.775 INFO:teuthology.orchestra.run.smithi168.stdout: "zabbix_host": { 2020-03-28T06:19:19.775 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "zabbix_host", 2020-03-28T06:19:19.775 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.775 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.775 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.776 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "None", 2020-03-28T06:19:19.776 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.776 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.776 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.776 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.776 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.776 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.777 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.777 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.777 INFO:teuthology.orchestra.run.smithi168.stdout: "zabbix_port": { 2020-03-28T06:19:19.777 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "zabbix_port", 2020-03-28T06:19:19.777 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "int", 2020-03-28T06:19:19.777 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.777 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "10051", 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.778 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.779 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.779 INFO:teuthology.orchestra.run.smithi168.stdout: "zabbix_sender": { 2020-03-28T06:19:19.779 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "zabbix_sender", 2020-03-28T06:19:19.779 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "str", 2020-03-28T06:19:19.779 INFO:teuthology.orchestra.run.smithi168.stdout: "level": "advanced", 2020-03-28T06:19:19.779 INFO:teuthology.orchestra.run.smithi168.stdout: "flags": 0, 2020-03-28T06:19:19.779 INFO:teuthology.orchestra.run.smithi168.stdout: "default_value": "/usr/bin/zabbix_sender", 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: "min": "", 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: "max": "", 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: "enum_allowed": [], 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: "desc": "", 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: "long_desc": "", 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: "see_also": [] 2020-03-28T06:19:19.780 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.781 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.781 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.781 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-03-28T06:19:19.781 INFO:teuthology.orchestra.run.smithi168.stdout: "services": {}, 2020-03-28T06:19:19.781 INFO:teuthology.orchestra.run.smithi168.stdout: "always_on_modules": { 2020-03-28T06:19:19.781 INFO:teuthology.orchestra.run.smithi168.stdout: "nautilus": [ 2020-03-28T06:19:19.781 INFO:teuthology.orchestra.run.smithi168.stdout: "balancer", 2020-03-28T06:19:19.782 INFO:teuthology.orchestra.run.smithi168.stdout: "crash", 2020-03-28T06:19:19.782 INFO:teuthology.orchestra.run.smithi168.stdout: "devicehealth", 2020-03-28T06:19:19.782 INFO:teuthology.orchestra.run.smithi168.stdout: "orchestrator_cli", 2020-03-28T06:19:19.782 INFO:teuthology.orchestra.run.smithi168.stdout: "progress", 2020-03-28T06:19:19.782 INFO:teuthology.orchestra.run.smithi168.stdout: "rbd_support", 2020-03-28T06:19:19.782 INFO:teuthology.orchestra.run.smithi168.stdout: "status", 2020-03-28T06:19:19.782 INFO:teuthology.orchestra.run.smithi168.stdout: "volumes" 2020-03-28T06:19:19.783 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-03-28T06:19:19.783 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-03-28T06:19:19.783 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.783 INFO:teuthology.orchestra.run.smithi168.stdout: "servicemap": { 2020-03-28T06:19:19.783 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 1, 2020-03-28T06:19:19.783 INFO:teuthology.orchestra.run.smithi168.stdout: "modified": "0.000000", 2020-03-28T06:19:19.783 INFO:teuthology.orchestra.run.smithi168.stdout: "services": {} 2020-03-28T06:19:19.784 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-03-28T06:19:19.784 INFO:teuthology.orchestra.run.smithi168.stdout: "progress_events": {} 2020-03-28T06:19:19.784 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-03-28T06:19:19.786 INFO:tasks.ceph.ceph_manager.c2:{u'progress_events': {}, u'election_epoch': 3, u'quorum_age': 37, u'mgrmap': {u'available': True, u'active_change': u'2020-03-28 06:18:45.089663', u'available_modules': [{u'module_options': {u'smtp_port': {u'default_value': u'465', u'long_desc': u'', u'name': u'smtp_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'SMTP port'}, u'smtp_destination': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_destination', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Email address to send alerts to'}, u'smtp_sender': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_sender', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'SMTP envelope sender'}, u'interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'How frequently to reexamine health status'}, u'smtp_host': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_host', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'SMTP server'}, u'smtp_ssl': {u'default_value': u'True', u'long_desc': u'', u'name': u'smtp_ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'Use SSL to connect to SMTP server'}, u'smtp_from_name': {u'default_value': u'Ceph', u'long_desc': u'', u'name': u'smtp_from_name', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Email From: name'}, u'smtp_user': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_user', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'User to authenticate as'}, u'smtp_password': {u'default_value': u'', u'long_desc': u'', u'name': u'smtp_password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Password to authenticate with'}}, u'name': u'alerts', u'error_string': u'', u'can_run': True}, {u'module_options': {u'username': {u'default_value': u'', u'long_desc': u'', u'name': u'username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'password': {u'default_value': u'', u'long_desc': u'', u'name': u'password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_url': {u'default_value': u'', u'long_desc': u'', u'name': u'server_url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'verify_server': {u'default_value': u'', u'long_desc': u'', u'name': u'verify_server', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'ansible', u'error_string': u'', u'can_run': True}, {u'module_options': {u'min_score': {u'default_value': u'0', u'long_desc': u'', u'name': u'min_score', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'float', u'tags': [], u'desc': u'minimum score, below which no optimization is attempted'}, u'end_weekday': {u'default_value': u'7', u'long_desc': u'0 or 7 = Sunday, 1 = Monday, etc.', u'name': u'end_weekday', u'min': u'0', u'max': u'7', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'Restrict automatic balancing to days of the week earlier than this'}, u'pool_ids': {u'default_value': u'', u'long_desc': u'', u'name': u'pool_ids', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'pools which the automatic balancing will be limited to'}, u'sleep_interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to wake up and attempt optimization'}, u'begin_time': {u'default_value': u'0000', u'long_desc': u'This is a time of day in the format HHMM.', u'name': u'begin_time', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'beginning time of day to automatically balance'}, u'crush_compat_metrics': {u'default_value': u'pgs,objects,bytes', u'long_desc': u'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', u'name': u'crush_compat_metrics', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'metrics with which to calculate OSD utilization'}, u'begin_weekday': {u'default_value': u'0', u'long_desc': u'0 or 7 = Sunday, 1 = Monday, etc.', u'name': u'begin_weekday', u'min': u'0', u'max': u'7', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'Restrict automatic balancing to this day of the week or later'}, u'upmap_max_iterations': {u'default_value': u'10', u'long_desc': u'', u'name': u'upmap_max_iterations', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'maximum upmap optimization iterations'}, u'upmap_max_deviation': {u'default_value': u'5', u'long_desc': u'If the number of PGs are within this count then no optimization is attempted', u'name': u'upmap_max_deviation', u'min': u'1', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'deviation below which no optimization is attempted'}, u'end_time': {u'default_value': u'2400', u'long_desc': u'This is a time of day in the format HHMM.', u'name': u'end_time', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'ending time of day to automatically balance'}, u'crush_compat_step': {u'default_value': u'0.5', u'long_desc': u'.99 is very aggressive, .01 is less aggressive', u'name': u'crush_compat_step', u'min': u'0.001', u'max': u'0.999', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'float', u'tags': [], u'desc': u'aggressiveness of optimization'}, u'crush_compat_max_iterations': {u'default_value': u'25', u'long_desc': u'', u'name': u'crush_compat_max_iterations', u'min': u'1', u'max': u'250', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'uint', u'tags': [], u'desc': u'maximum number of iterations to attempt optimization'}, u'active': {u'default_value': u'False', u'long_desc': u'', u'name': u'active', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'automatically balance PGs across cluster'}, u'mode': {u'default_value': u'none', u'long_desc': u'', u'name': u'mode', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [u'crush-compat', u'none', u'upmap'], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'Balancer mode'}}, u'name': u'balancer', u'error_string': u'', u'can_run': True}, {u'module_options': {u'warn_recent_interval': {u'default_value': u'1209600', u'long_desc': u'', u'name': u'warn_recent_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'time interval in which to warn about recent crashes'}, u'retain_interval': {u'default_value': u'31536000', u'long_desc': u'', u'name': u'retain_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how long to retain crashes before pruning them'}}, u'name': u'crash', u'error_string': u'', u'can_run': True}, {u'module_options': {u'FEATURE_TOGGLE_cephfs': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_cephfs', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_rbd': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_rbd', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'AUDIT_API_ENABLED': {u'default_value': u'False', u'long_desc': u'', u'name': u'AUDIT_API_ENABLED', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'RGW_API_SECRET_KEY': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_SECRET_KEY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ENABLE_BROWSABLE_API': {u'default_value': u'True', u'long_desc': u'', u'name': u'ENABLE_BROWSABLE_API', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'crt_file': {u'default_value': u'', u'long_desc': u'', u'name': u'crt_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'PROMETHEUS_API_HOST': {u'default_value': u'', u'long_desc': u'', u'name': u'PROMETHEUS_API_HOST', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'REST_REQUESTS_TIMEOUT': {u'default_value': u'45', u'long_desc': u'', u'name': u'REST_REQUESTS_TIMEOUT', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'GRAFANA_API_USERNAME': {u'default_value': u'admin', u'long_desc': u'', u'name': u'GRAFANA_API_USERNAME', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'standby_behaviour': {u'default_value': u'redirect', u'long_desc': u'', u'name': u'standby_behaviour', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [u'error', u'redirect'], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'url_prefix': {u'default_value': u'', u'long_desc': u'', u'name': u'url_prefix', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'password': {u'default_value': u'', u'long_desc': u'', u'name': u'password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_ADMIN_RESOURCE': {u'default_value': u'admin', u'long_desc': u'', u'name': u'RGW_API_ADMIN_RESOURCE', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_PORT': {u'default_value': u'80', u'long_desc': u'', u'name': u'RGW_API_PORT', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'jwt_token_ttl': {u'default_value': u'28800', u'long_desc': u'', u'name': u'jwt_token_ttl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'GRAFANA_API_URL': {u'default_value': u'', u'long_desc': u'', u'name': u'GRAFANA_API_URL', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ALERTMANAGER_API_HOST': {u'default_value': u'', u'long_desc': u'', u'name': u'ALERTMANAGER_API_HOST', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_iscsi': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_iscsi', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'server_port': {u'default_value': u'8080', u'long_desc': u'', u'name': u'server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {u'default_value': u'', u'long_desc': u'', u'name': u'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_ACCESS_KEY': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_ACCESS_KEY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'AUDIT_API_LOG_PAYLOAD': {u'default_value': u'True', u'long_desc': u'', u'name': u'AUDIT_API_LOG_PAYLOAD', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'RGW_API_HOST': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_HOST', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_SCHEME': {u'default_value': u'http', u'long_desc': u'', u'name': u'RGW_API_SCHEME', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'username': {u'default_value': u'', u'long_desc': u'', u'name': u'username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'RGW_API_SSL_VERIFY': {u'default_value': u'True', u'long_desc': u'', u'name': u'RGW_API_SSL_VERIFY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'RGW_API_USER_ID': {u'default_value': u'', u'long_desc': u'', u'name': u'RGW_API_USER_ID', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'GRAFANA_UPDATE_DASHBOARDS': {u'default_value': u'False', u'long_desc': u'', u'name': u'GRAFANA_UPDATE_DASHBOARDS', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_rgw': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_rgw', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'GRAFANA_API_PASSWORD': {u'default_value': u'admin', u'long_desc': u'', u'name': u'GRAFANA_API_PASSWORD', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ssl': {u'default_value': u'True', u'long_desc': u'', u'name': u'ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'standby_error_status_code': {u'default_value': u'500', u'long_desc': u'', u'name': u'standby_error_status_code', u'min': u'400', u'max': u'599', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'ISCSI_API_SSL_VERIFICATION': {u'default_value': u'True', u'long_desc': u'', u'name': u'ISCSI_API_SSL_VERIFICATION', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'ssl_server_port': {u'default_value': u'8443', u'long_desc': u'', u'name': u'ssl_server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'server_addr': {u'default_value': u'::', u'long_desc': u'', u'name': u'server_addr', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'FEATURE_TOGGLE_mirroring': {u'default_value': u'True', u'long_desc': u'', u'name': u'FEATURE_TOGGLE_mirroring', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'debug': {u'default_value': u'False', u'long_desc': u'', u'name': u'debug', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'Enable/disable debug options'}, u'key_file': {u'default_value': u'', u'long_desc': u'', u'name': u'key_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'GRAFANA_API_SSL_VERIFY': {u'default_value': u'True', u'long_desc': u'', u'name': u'GRAFANA_API_SSL_VERIFY', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}}, u'name': u'dashboard', u'error_string': u'', u'can_run': True}, {u'module_options': {u'salt_api_username': {u'default_value': u'', u'long_desc': u'', u'name': u'salt_api_username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'salt_api_password': {u'default_value': u'', u'long_desc': u'', u'name': u'salt_api_password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'salt_api_eauth': {u'default_value': u'sharedsecret', u'long_desc': u'', u'name': u'salt_api_eauth', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'salt_api_url': {u'default_value': u'', u'long_desc': u'', u'name': u'salt_api_url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'deepsea', u'error_string': u'', u'can_run': True}, {u'module_options': {u'pool_name': {u'default_value': u'device_health_metrics', u'long_desc': u'', u'name': u'pool_name', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'name of pool in which to store device health metrics'}, u'self_heal': {u'default_value': u'True', u'long_desc': u'', u'name': u'self_heal', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'preemptively heal cluster around devices that may fail'}, u'sleep_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to wake up and check device health'}, u'scrape_frequency': {u'default_value': u'86400', u'long_desc': u'', u'name': u'scrape_frequency', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to scrape device health metrics'}, u'enable_monitoring': {u'default_value': u'False', u'long_desc': u'', u'name': u'enable_monitoring', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'monitor device health metrics'}, u'retention_period': {u'default_value': u'15552000', u'long_desc': u'', u'name': u'retention_period', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how long to retain device health metrics'}, u'mark_out_threshold': {u'default_value': u'2419200', u'long_desc': u'', u'name': u'mark_out_threshold', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'automatically mark OSD if it may fail before this long'}, u'warn_threshold': {u'default_value': u'7257600', u'long_desc': u'', u'name': u'warn_threshold', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'raise health warning if OSD may fail before this long'}}, u'name': u'devicehealth', u'error_string': u'', u'can_run': True}, {u'module_options': {u'diskprediction_cert_context': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_cert_context', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_default_authority': {u'default_value': u'localhost', u'long_desc': u'', u'name': u'diskprediction_default_authority', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_ssl_target_name_override': {u'default_value': u'localhost', u'long_desc': u'', u'name': u'diskprediction_ssl_target_name_override', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'sleep_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_retrieve_prediction_interval': {u'default_value': u'43200', u'long_desc': u'', u'name': u'diskprediction_retrieve_prediction_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_upload_smart_interval': {u'default_value': u'43200', u'long_desc': u'', u'name': u'diskprediction_upload_smart_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_user': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_user', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_upload_metrics_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'diskprediction_upload_metrics_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_password': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_port': {u'default_value': u'31400', u'long_desc': u'', u'name': u'diskprediction_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'diskprediction_server': {u'default_value': u'', u'long_desc': u'', u'name': u'diskprediction_server', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'diskprediction_cloud', u'error_string': u'', u'can_run': True}, {u'module_options': {u'predict_interval': {u'default_value': u'86400', u'long_desc': u'', u'name': u'predict_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'sleep_interval': {u'default_value': u'600', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'diskprediction_local', u'error_string': u'', u'can_run': True}, {u'module_options': {u'username': {u'default_value': u'None', u'long_desc': u'', u'name': u'username', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'interval': {u'default_value': u'30', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'database': {u'default_value': u'ceph', u'long_desc': u'', u'name': u'database', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'hostname': {u'default_value': u'None', u'long_desc': u'', u'name': u'hostname', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'batch_size': {u'default_value': u'5000', u'long_desc': u'', u'name': u'batch_size', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ssl': {u'default_value': u'false', u'long_desc': u'', u'name': u'ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'verify_ssl': {u'default_value': u'true', u'long_desc': u'', u'name': u'verify_ssl', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'threads': {u'default_value': u'5', u'long_desc': u'', u'name': u'threads', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'password': {u'default_value': u'None', u'long_desc': u'', u'name': u'password', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'port': {u'default_value': u'8086', u'long_desc': u'', u'name': u'port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'influx', u'error_string': u'influxdb python module not found', u'can_run': False}, {u'module_options': {}, u'name': u'insights', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'iostat', u'error_string': u'', u'can_run': True}, {u'module_options': {u'failure_domain': {u'default_value': u'host', u'long_desc': u'', u'name': u'failure_domain', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'failure domain for any created local pool'}, u'subtree': {u'default_value': u'rack', u'long_desc': u'', u'name': u'subtree', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'CRUSH level for which to create a local pool'}, u'min_size': {u'default_value': u'', u'long_desc': u'', u'name': u'min_size', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'default min_size for any created local pool'}, u'prefix': {u'default_value': u'', u'long_desc': u'', u'name': u'prefix', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u'name prefix for any created local pool'}, u'pg_num': {u'default_value': u'128', u'long_desc': u'', u'name': u'pg_num', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'default pg_num for any created local pool'}, u'num_rep': {u'default_value': u'3', u'long_desc': u'', u'name': u'num_rep', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'default replica count for any created local pool'}}, u'name': u'localpool', u'error_string': u'', u'can_run': True}, {u'module_options': {u'orchestrator': {u'default_value': u'', u'long_desc': u'', u'name': u'orchestrator', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'orchestrator_cli', u'error_string': u'', u'can_run': True}, {u'module_options': {u'sleep_interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'sleep_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'pg_autoscaler', u'error_string': u'', u'can_run': True}, {u'module_options': {u'persist_interval': {u'default_value': u'5', u'long_desc': u'', u'name': u'persist_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u'how frequently to persist completed events'}, u'max_completed_events': {u'default_value': u'50', u'long_desc': u'', u'name': u'max_completed_events', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 1, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u'number of past completed events to remember'}}, u'name': u'progress', u'error_string': u'', u'can_run': True}, {u'module_options': {u'rbd_stats_pools_refresh_interval': {u'default_value': u'', u'long_desc': u'', u'name': u'rbd_stats_pools_refresh_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rbd_stats_pools': {u'default_value': u'', u'long_desc': u'', u'name': u'rbd_stats_pools', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_port': {u'default_value': u'', u'long_desc': u'', u'name': u'server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_addr': {u'default_value': u'', u'long_desc': u'', u'name': u'server_addr', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'scrape_interval': {u'default_value': u'', u'long_desc': u'', u'name': u'scrape_interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'prometheus', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'rbd_support', u'error_string': u'', u'can_run': True}, {u'module_options': {u'key_file': {u'default_value': u'', u'long_desc': u'', u'name': u'key_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_port': {u'default_value': u'', u'long_desc': u'', u'name': u'server_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'server_addr': {u'default_value': u'', u'long_desc': u'', u'name': u'server_addr', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'restful', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'rook', u'error_string': u'', u'can_run': True}, {u'module_options': {u'testkey': {u'default_value': u'', u'long_desc': u'', u'name': u'testkey', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rwoption6': {u'default_value': u'True', u'long_desc': u'', u'name': u'rwoption6', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'rwoption5': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption5', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'roption2': {u'default_value': u'xyz', u'long_desc': u'', u'name': u'roption2', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'roption1': {u'default_value': u'', u'long_desc': u'', u'name': u'roption1', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rwoption4': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption4', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'rwoption3': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption3', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'float', u'tags': [], u'desc': u''}, u'rwoption2': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption2', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'rwoption1': {u'default_value': u'', u'long_desc': u'', u'name': u'rwoption1', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'testnewline': {u'default_value': u'', u'long_desc': u'', u'name': u'testnewline', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'testlkey': {u'default_value': u'', u'long_desc': u'', u'name': u'testlkey', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'selftest', u'error_string': u'', u'can_run': True}, {u'module_options': {u'inventory_cache_timeout_min': {u'default_value': u'', u'long_desc': u'', u'name': u'inventory_cache_timeout_min', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'ssh_config_file': {u'default_value': u'', u'long_desc': u'', u'name': u'ssh_config_file', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'ssh', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'status', u'error_string': u'', u'can_run': True}, {u'module_options': {u'interval': {u'default_value': u'15', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u''}, u'address': {u'default_value': u'unixgram:///tmp/telegraf.sock', u'long_desc': u'', u'name': u'address', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'telegraf', u'error_string': u'', u'can_run': True}, {u'module_options': {u'channel_crash': {u'default_value': u'True', u'long_desc': u'', u'name': u'channel_crash', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'channel_basic': {u'default_value': u'True', u'long_desc': u'', u'name': u'channel_basic', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u'Share basic cluster information (size, version)'}, u'description': {u'default_value': u'None', u'long_desc': u'', u'name': u'description', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'url': {u'default_value': u'https://telemetry.ceph.com/report', u'long_desc': u'', u'name': u'url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'enabled': {u'default_value': u'False', u'long_desc': u'', u'name': u'enabled', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'interval': {u'default_value': u'24', u'long_desc': u'', u'name': u'interval', u'min': u'8', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'channel_device': {u'default_value': u'True', u'long_desc': u'', u'name': u'channel_device', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'channel_ident': {u'default_value': u'False', u'long_desc': u'', u'name': u'channel_ident', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'device_url': {u'default_value': u'https://telemetry.ceph.com/device', u'long_desc': u'', u'name': u'device_url', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'contact': {u'default_value': u'None', u'long_desc': u'', u'name': u'contact', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'leaderboard': {u'default_value': u'False', u'long_desc': u'', u'name': u'leaderboard', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'bool', u'tags': [], u'desc': u''}, u'organization': {u'default_value': u'None', u'long_desc': u'', u'name': u'organization', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'last_opt_revision': {u'default_value': u'1', u'long_desc': u'', u'name': u'last_opt_revision', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'proxy': {u'default_value': u'None', u'long_desc': u'', u'name': u'proxy', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}}, u'name': u'telemetry', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'test_orchestrator', u'error_string': u'', u'can_run': True}, {u'module_options': {}, u'name': u'volumes', u'error_string': u'', u'can_run': True}, {u'module_options': {u'zabbix_host': {u'default_value': u'None', u'long_desc': u'', u'name': u'zabbix_host', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'identifier': {u'default_value': u'', u'long_desc': u'', u'name': u'identifier', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'zabbix_sender': {u'default_value': u'/usr/bin/zabbix_sender', u'long_desc': u'', u'name': u'zabbix_sender', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'str', u'tags': [], u'desc': u''}, u'zabbix_port': {u'default_value': u'10051', u'long_desc': u'', u'name': u'zabbix_port', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'int', u'tags': [], u'desc': u''}, u'interval': {u'default_value': u'60', u'long_desc': u'', u'name': u'interval', u'min': u'', u'max': u'', u'level': u'advanced', u'enum_allowed': [], u'flags': 0, u'see_also': [], u'type': u'secs', u'tags': [], u'desc': u''}}, u'name': u'zabbix', u'error_string': u'', u'can_run': True}], u'always_on_modules': {u'nautilus': [u'balancer', u'crash', u'devicehealth', u'orchestrator_cli', u'progress', u'rbd_support', u'status', u'volumes']}, u'modules': [u'iostat', u'restful'], u'active_name': u'x', u'epoch': 3, u'active_addr': u'172.21.15.168:6801/8270', u'standbys': [], u'active_addrs': {u'addrvec': [{u'nonce': 8270, u'type': u'v2', u'addr': u'172.21.15.168:6800'}, {u'nonce': 8270, u'type': u'v1', u'addr': u'172.21.15.168:6801'}]}, u'services': {}, u'active_gid': 4098}, u'monmap': {u'features': {u'optional': [], u'persistent': [u'kraken', u'luminous', u'mimic', u'osdmap-prune', u'nautilus']}, u'created': u'2020-03-28 06:18:17.802186', u'min_mon_release': 14, u'modified': u'2020-03-28 06:18:17.802186', u'epoch': 1, u'mons': [{u'public_addr': u'172.21.15.168:6789/0', u'addr': u'172.21.15.168:6789/0', u'name': u'a', u'rank': 0, u'public_addrs': {u'addrvec': [{u'nonce': 0, u'type': u'v2', u'addr': u'172.21.15.168:3300'}, {u'nonce': 0, u'type': u'v1', u'addr': u'172.21.15.168:6789'}]}}], u'min_mon_release_name': u'nautilus', u'fsid': u'2cc67b71-ad7c-4570-92b4-a0ade70c85c2'}, u'health': {u'status': u'HEALTH_OK', u'checks': {}}, u'pgmap': {u'bytes_total': 322122547200, u'num_pgs': 8, u'data_bytes': 0, u'bytes_used': 3221594112, u'num_objects': 0, u'num_pools': 1, u'pgs_by_state': [{u'count': 8, u'state_name': u'active+clean'}], u'bytes_avail': 318900953088}, u'quorum_names': [u'a'], u'fsmap': {u'up:standby': 0, u'epoch': 1, u'by_rank': []}, u'osdmap': {u'osdmap': {u'num_up_osds': 3, u'epoch': 13, u'num_in_osds': 3, u'num_osds': 3, u'num_remapped_pgs': 0}}, u'servicemap': {u'services': {}, u'epoch': 1, u'modified': u'0.000000'}, u'fsid': u'2cc67b71-ad7c-4570-92b4-a0ade70c85c2', u'quorum': [0]} 2020-03-28T06:19:19.788 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-03-28T06:19:19.788 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-03-28T06:19:19.789 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 health 2020-03-28T06:19:20.654 INFO:teuthology.misc.health.smithi168.stdout:HEALTH_OK 2020-03-28T06:19:20.670 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2020-03-28T06:19:20.671 INFO:teuthology.run_tasks:Running task rgw... 2020-03-28T06:19:20.683 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c1.client.1': {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c2.client.1': {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}, 'c2.client.0': {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']}} 2020-03-28T06:19:20.684 DEBUG:tasks.rgw:client list is ['c2.client.1', 'c2.client.0', 'c1.client.0', 'c1.client.1'] 2020-03-28T06:19:20.684 INFO:tasks.rgw:Creating data pools 2020-03-28T06:19:20.684 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-03-28T06:19:20.684 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-03-28T06:19:21.883 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.data' created 2020-03-28T06:19:21.903 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-03-28T06:19:22.909 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-03-28T06:19:22.927 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-03-28T06:19:23.972 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.index' created 2020-03-28T06:19:23.990 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-03-28T06:19:25.012 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-03-28T06:19:25.026 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-03-28T06:19:25.026 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-03-28T06:19:25.775 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.data' already exists 2020-03-28T06:19:25.788 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-03-28T06:19:27.363 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-03-28T06:19:27.375 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-03-28T06:19:28.091 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.index' already exists 2020-03-28T06:19:28.108 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-03-28T06:19:29.530 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-03-28T06:19:29.554 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-03-28T06:19:29.555 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-03-28T06:19:30.640 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.data' created 2020-03-28T06:19:30.661 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-03-28T06:19:31.676 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-03-28T06:19:31.695 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-03-28T06:19:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.index' created 2020-03-28T06:19:32.764 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-03-28T06:19:33.826 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-03-28T06:19:33.840 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-03-28T06:19:33.840 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-03-28T06:19:34.529 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.data' already exists 2020-03-28T06:19:34.543 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-03-28T06:19:36.152 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-03-28T06:19:36.162 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-03-28T06:19:36.915 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.index' already exists 2020-03-28T06:19:36.927 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-03-28T06:19:38.366 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-03-28T06:19:38.388 DEBUG:tasks.rgw:Pools created 2020-03-28T06:19:38.388 INFO:tasks.rgw:Configuring compression type = random 2020-03-28T06:19:38.389 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-03-28T06:19:38.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-03-28T06:19:38.389 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-03-28T06:19:38.666 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:38.668 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:40.810 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:40.812 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:40.810 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:40.812 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:41.134 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:41.136 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:41.134 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:41.136 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:41.148 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:41.150 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:41.148 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:41.150 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:41.383 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:41.385 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:43.365 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.367 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:43.372 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.374 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:43.372 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.374 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:43.378 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.380 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:43.378 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.380 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:43.412 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.414 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:43.412 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.414 7f39478a7840 10 cannot find current period zonegroup using local zonegroup 2020-03-28T06:19:43.412 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.414 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:43.418 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.420 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:43.418 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.420 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:43.423 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.425 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:43.424 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.425 7f39478a7840 10 Creating default zonegroup 2020-03-28T06:19:43.424 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.425 7f39478a7840 20 get_system_obj_state: rctx=0x7ffe673e99d0 obj=:.pools.avail state=0x5596a169f880 s->prefetch_data=0 2020-03-28T06:19:43.426 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.428 7f39478a7840 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-03-28T06:19:43.608 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.610 7f39478a7840 10 WARNING: store->list_zones() returned r=-2 2020-03-28T06:19:43.609 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.610 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:43.612 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:43.614 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:44.297 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.299 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:44.310 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.312 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:44.310 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.312 7f39478a7840 10 could not read realm id: (2) No such file or directory 2020-03-28T06:19:44.310 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.312 7f39478a7840 10 WARNING: failed to set zone as default, r=-22 2020-03-28T06:19:44.310 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.312 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:44.319 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.321 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:44.769 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.771 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:44.825 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.826 7f39478a7840 20 rados_obj.operate() r=0 bl.length=805 2020-03-28T06:19:44.825 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.827 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:44.876 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.878 7f39478a7840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:19:44.876 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.878 7f39478a7840 20 zonegroup default 2020-03-28T06:19:44.876 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.878 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:44.886 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.888 7f39478a7840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:44.886 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.888 7f39478a7840 10 Cannot find current period zone using local zone 2020-03-28T06:19:44.887 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.888 7f39478a7840 10 Using default name default 2020-03-28T06:19:44.887 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.888 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:44.942 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.944 7f39478a7840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:44.942 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.944 7f39478a7840 20 rados->read ofs=0 len=0 2020-03-28T06:19:44.958 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.960 7f39478a7840 20 rados_obj.operate() r=0 bl.length=805 2020-03-28T06:19:44.958 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:44.960 7f39478a7840 20 zone default 2020-03-28T06:19:48.226 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.228 7f39478a7840 20 add_watcher() i=0 2020-03-28T06:19:48.226 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.228 7f39478a7840 20 add_watcher() i=1 2020-03-28T06:19:48.226 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.228 7f39478a7840 20 add_watcher() i=2 2020-03-28T06:19:48.226 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.228 7f39478a7840 20 add_watcher() i=3 2020-03-28T06:19:48.227 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.228 7f39478a7840 20 add_watcher() i=4 2020-03-28T06:19:48.227 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.228 7f39478a7840 20 add_watcher() i=5 2020-03-28T06:19:48.240 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.242 7f39478a7840 20 add_watcher() i=6 2020-03-28T06:19:48.783 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.783 7f39478a7840 20 add_watcher() i=7 2020-03-28T06:19:48.783 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.783 7f39478a7840 2 all 8 watchers are set, enabling cache 2020-03-28T06:19:48.813 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:48.816 7f3908ff9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:19:50.427 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:50.429 7f39478a7840 10 rgw_init_ioctx warning: failed to set pg_autoscale_bias on default.rgw.meta 2020-03-28T06:19:50.430 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:50.432 7f39478a7840 10 rgw_init_ioctx warning: failed to set pg_num_min on default.rgw.meta 2020-03-28T06:19:52.537 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:52.539 7f39478a7840 10 rgw_init_ioctx warning: failed to set pg_autoscale_bias on default.rgw.log 2020-03-28T06:19:52.542 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:52.544 7f39478a7840 10 rgw_init_ioctx warning: failed to set pg_num_min on default.rgw.log 2020-03-28T06:19:52.543 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:52.545 7f38eb7de700 20 reqs_thread_entry: start 2020-03-28T06:19:52.544 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:52.546 7f38e97da700 20 reqs_thread_entry: start 2020-03-28T06:19:52.544 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:52.546 7f39478a7840 20 init_complete bucket index max shards: 0 2020-03-28T06:19:52.544 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:52.546 7f38577fe700 20 reqs_thread_entry: start 2020-03-28T06:19:56.765 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:56.767 7f39478a7840 20 remove_watcher() i=0 2020-03-28T06:19:56.765 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:56.767 7f39478a7840 2 removed watcher, disabling cache 2020-03-28T06:19:56.825 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:56.827 7f39478a7840 20 remove_watcher() i=1 2020-03-28T06:19:56.901 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:56.903 7f39478a7840 20 remove_watcher() i=2 2020-03-28T06:19:56.950 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:56.953 7f39478a7840 20 remove_watcher() i=3 2020-03-28T06:19:57.013 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.015 7f39478a7840 20 remove_watcher() i=4 2020-03-28T06:19:57.069 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.072 7f39478a7840 20 remove_watcher() i=5 2020-03-28T06:19:57.104 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.107 7f39478a7840 20 remove_watcher() i=6 2020-03-28T06:19:57.183 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.185 7f39478a7840 20 remove_watcher() i=7 2020-03-28T06:19:57.193 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T06:19:57.194 DEBUG:tasks.util.rgw: json result: [] 2020-03-28T06:19:57.194 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:19:57.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:19:57.194 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-03-28T06:19:57.459 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.461 7f6805ffb700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:19:57.459 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.461 7f68057fa700 20 reqs_thread_entry: start 2020-03-28T06:19:57.459 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.461 7f6841005840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.471 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.473 7f6841005840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:57.471 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.473 7f6841005840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.482 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.484 7f6841005840 20 rados_obj.operate() r=0 bl.length=805 2020-03-28T06:19:57.482 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.484 7f6841005840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.487 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.489 7f6841005840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:57.487 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.489 7f6841005840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.494 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.497 7f6841005840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:57.494 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.497 7f6841005840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.506 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.508 7f6841005840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:19:57.603 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"5d5948fb-3b8e-4188-99f4-bb7488595f7a","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":""} 2020-03-28T06:19:57.604 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'metadata_heap': u'', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'user_email_pool': u'default.rgw.meta:users.email', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'placement_pools': [{u'val': {u'storage_classes': {u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'realm_id': u'', u'id': u'5d5948fb-3b8e-4188-99f4-bb7488595f7a', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2020-03-28T06:19:57.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-03-28T06:19:57.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-03-28T06:19:57.605 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-03-28T06:19:57.791 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.793 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.814 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.816 7febceea8840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:57.814 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.816 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.824 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.827 7febceea8840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:57.825 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.827 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.831 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.833 7febceea8840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:57.831 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.833 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:57.843 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:57.846 7febceea8840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:58.131 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.133 7febceea8840 20 RGWRados::pool_iterate: got zone_info.5d5948fb-3b8e-4188-99f4-bb7488595f7a 2020-03-28T06:19:58.131 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.133 7febceea8840 20 RGWRados::pool_iterate: got zonegroup_info.437fc0e1-5efa-4cb0-981b-97c68ec6cf36 2020-03-28T06:19:58.131 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.133 7febceea8840 20 RGWRados::pool_iterate: got zone_names.default 2020-03-28T06:19:58.131 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.133 7febceea8840 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-03-28T06:19:58.131 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.133 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.143 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.145 7febceea8840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:58.143 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.145 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.154 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.156 7febceea8840 20 rados_obj.operate() r=0 bl.length=821 2020-03-28T06:19:58.154 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.156 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.168 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.170 7febceea8840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:58.168 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.170 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.181 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.183 7febceea8840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:19:58.181 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.183 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.190 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.192 7febceea8840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:58.190 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.192 7febceea8840 10 cannot find current period zonegroup using local zonegroup 2020-03-28T06:19:58.190 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.192 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.198 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.200 7febceea8840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:58.198 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.200 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.209 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.212 7febceea8840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:58.209 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.212 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.223 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.225 7febceea8840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:19:58.223 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.225 7febceea8840 20 zonegroup default 2020-03-28T06:19:58.223 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.225 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.229 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.231 7febceea8840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:58.229 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.231 7febceea8840 10 Cannot find current period zone using local zone 2020-03-28T06:19:58.229 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.231 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.235 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.238 7febceea8840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:58.235 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.238 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.245 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.247 7febceea8840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:58.245 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.247 7febceea8840 20 rados->read ofs=0 len=0 2020-03-28T06:19:58.257 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.260 7febceea8840 20 rados_obj.operate() r=0 bl.length=821 2020-03-28T06:19:58.258 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.260 7febceea8840 20 zone default 2020-03-28T06:19:58.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.577 7febceea8840 20 add_watcher() i=0 2020-03-28T06:19:58.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.577 7febceea8840 20 add_watcher() i=1 2020-03-28T06:19:58.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.577 7febceea8840 20 add_watcher() i=2 2020-03-28T06:19:58.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.577 7febceea8840 20 add_watcher() i=3 2020-03-28T06:19:58.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.577 7febceea8840 20 add_watcher() i=4 2020-03-28T06:19:58.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.577 7febceea8840 20 add_watcher() i=5 2020-03-28T06:19:58.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.577 7febceea8840 20 add_watcher() i=6 2020-03-28T06:19:58.602 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.604 7febceea8840 20 add_watcher() i=7 2020-03-28T06:19:58.602 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.604 7febceea8840 2 all 8 watchers are set, enabling cache 2020-03-28T06:19:58.635 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.637 7feb93fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:19:58.643 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.646 7feb7a7e4700 20 reqs_thread_entry: start 2020-03-28T06:19:58.644 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.646 7feb3bfff700 20 reqs_thread_entry: start 2020-03-28T06:19:58.644 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.646 7febceea8840 20 init_complete bucket index max shards: 0 2020-03-28T06:19:58.644 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.646 7feb3a7fc700 20 reqs_thread_entry: start 2020-03-28T06:19:58.877 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.879 7febceea8840 20 remove_watcher() i=0 2020-03-28T06:19:58.877 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.879 7febceea8840 2 removed watcher, disabling cache 2020-03-28T06:19:58.906 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.909 7febceea8840 20 remove_watcher() i=1 2020-03-28T06:19:58.943 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.946 7febceea8840 20 remove_watcher() i=2 2020-03-28T06:19:58.991 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:58.993 7febceea8840 20 remove_watcher() i=3 2020-03-28T06:19:59.030 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.032 7febceea8840 20 remove_watcher() i=4 2020-03-28T06:19:59.065 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.067 7febceea8840 20 remove_watcher() i=5 2020-03-28T06:19:59.102 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.104 7febceea8840 20 remove_watcher() i=6 2020-03-28T06:19:59.130 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.132 7febceea8840 20 remove_watcher() i=7 2020-03-28T06:19:59.136 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T06:19:59.136 DEBUG:tasks.util.rgw: json result: [] 2020-03-28T06:19:59.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:19:59.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:19:59.137 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-03-28T06:19:59.448 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.450 7f8af67fc700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:19:59.449 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.451 7f8af5ffb700 20 reqs_thread_entry: start 2020-03-28T06:19:59.449 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.451 7f8b313f3840 20 rados->read ofs=0 len=0 2020-03-28T06:19:59.465 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.467 7f8b313f3840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:59.465 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.467 7f8b313f3840 20 rados->read ofs=0 len=0 2020-03-28T06:19:59.483 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.485 7f8b313f3840 20 rados_obj.operate() r=0 bl.length=821 2020-03-28T06:19:59.483 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.485 7f8b313f3840 20 rados->read ofs=0 len=0 2020-03-28T06:19:59.488 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.490 7f8b313f3840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:19:59.488 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.490 7f8b313f3840 20 rados->read ofs=0 len=0 2020-03-28T06:19:59.501 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.503 7f8b313f3840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:19:59.501 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.503 7f8b313f3840 20 rados->read ofs=0 len=0 2020-03-28T06:19:59.514 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:19:59.516 7f8b313f3840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:19:59.561 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"5d5948fb-3b8e-4188-99f4-bb7488595f7a","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":""} 2020-03-28T06:19:59.561 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'metadata_heap': u'', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'user_email_pool': u'default.rgw.meta:users.email', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'placement_pools': [{u'val': {u'storage_classes': {u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'realm_id': u'', u'id': u'5d5948fb-3b8e-4188-99f4-bb7488595f7a', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2020-03-28T06:19:59.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-03-28T06:19:59.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-03-28T06:19:59.562 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-03-28T06:19:59.826 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:19:59.826 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:02.053 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.052 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:02.053 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.052 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:02.164 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.163 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:02.164 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.163 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:02.177 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.176 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:02.177 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.176 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:02.268 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.267 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:02.971 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.970 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:02.974 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.974 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:02.975 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.974 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:02.980 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.978 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:02.980 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:02.978 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:03.004 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.003 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:03.004 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.003 7fa0d766a840 10 cannot find current period zonegroup using local zonegroup 2020-03-28T06:20:03.004 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.003 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:03.011 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.011 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:03.012 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.011 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:03.015 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.014 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:03.015 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.014 7fa0d766a840 10 Creating default zonegroup 2020-03-28T06:20:03.016 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.014 7fa0d766a840 20 get_system_obj_state: rctx=0x7ffd6d07fed0 obj=:.pools.avail state=0x55aa07a93260 s->prefetch_data=0 2020-03-28T06:20:03.018 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.017 7fa0d766a840 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-03-28T06:20:03.564 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.563 7fa0d766a840 10 WARNING: store->list_zones() returned r=-2 2020-03-28T06:20:03.565 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.563 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:03.568 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:03.567 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:04.367 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.366 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:04.377 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.376 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:04.378 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.376 7fa0d766a840 10 could not read realm id: (2) No such file or directory 2020-03-28T06:20:04.378 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.377 7fa0d766a840 10 WARNING: failed to set zone as default, r=-22 2020-03-28T06:20:04.378 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.377 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:04.385 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.384 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:04.526 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.526 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:04.563 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.562 7fa0d766a840 20 rados_obj.operate() r=0 bl.length=805 2020-03-28T06:20:04.563 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.562 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:04.623 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.622 7fa0d766a840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:20:04.623 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.622 7fa0d766a840 20 zonegroup default 2020-03-28T06:20:04.624 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.622 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:04.633 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.632 7fa0d766a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:04.633 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.632 7fa0d766a840 10 Cannot find current period zone using local zone 2020-03-28T06:20:04.633 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.632 7fa0d766a840 10 Using default name default 2020-03-28T06:20:04.633 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.632 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:04.652 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.652 7fa0d766a840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:04.653 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.652 7fa0d766a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:04.667 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.666 7fa0d766a840 20 rados_obj.operate() r=0 bl.length=805 2020-03-28T06:20:04.667 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:04.666 7fa0d766a840 20 zone default 2020-03-28T06:20:09.334 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.332 7fa0d766a840 20 add_watcher() i=0 2020-03-28T06:20:09.334 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.332 7fa0d766a840 20 add_watcher() i=1 2020-03-28T06:20:09.335 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.332 7fa0d766a840 20 add_watcher() i=2 2020-03-28T06:20:09.335 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.332 7fa0d766a840 20 add_watcher() i=3 2020-03-28T06:20:09.335 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.332 7fa0d766a840 20 add_watcher() i=4 2020-03-28T06:20:09.335 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.332 7fa0d766a840 20 add_watcher() i=5 2020-03-28T06:20:09.335 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.332 7fa0d766a840 20 add_watcher() i=6 2020-03-28T06:20:09.461 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.460 7fa0d766a840 20 add_watcher() i=7 2020-03-28T06:20:09.462 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.460 7fa0d766a840 2 all 8 watchers are set, enabling cache 2020-03-28T06:20:09.506 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:09.505 7fa0b0ff9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:20:10.831 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:10.830 7fa0d766a840 10 rgw_init_ioctx warning: failed to set pg_autoscale_bias on default.rgw.meta 2020-03-28T06:20:10.834 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:10.833 7fa0d766a840 10 rgw_init_ioctx warning: failed to set pg_num_min on default.rgw.meta 2020-03-28T06:20:12.900 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:12.899 7fa0d766a840 10 rgw_init_ioctx warning: failed to set pg_autoscale_bias on default.rgw.log 2020-03-28T06:20:12.903 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:12.902 7fa0d766a840 10 rgw_init_ioctx warning: failed to set pg_num_min on default.rgw.log 2020-03-28T06:20:12.903 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:12.903 7fa07b7de700 20 reqs_thread_entry: start 2020-03-28T06:20:12.904 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:12.903 7fa0797da700 20 reqs_thread_entry: start 2020-03-28T06:20:12.904 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:12.903 7fa0d766a840 20 init_complete bucket index max shards: 0 2020-03-28T06:20:12.905 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:12.904 7fa0337fe700 20 reqs_thread_entry: start 2020-03-28T06:20:15.558 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:15.557 7fa0d766a840 20 remove_watcher() i=0 2020-03-28T06:20:15.558 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:15.557 7fa0d766a840 2 removed watcher, disabling cache 2020-03-28T06:20:15.689 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:15.689 7fa0d766a840 20 remove_watcher() i=1 2020-03-28T06:20:16.335 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.335 7fa0d766a840 20 remove_watcher() i=2 2020-03-28T06:20:16.416 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.415 7fa0d766a840 20 remove_watcher() i=3 2020-03-28T06:20:16.467 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.467 7fa0d766a840 20 remove_watcher() i=4 2020-03-28T06:20:16.502 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.501 7fa0d766a840 20 remove_watcher() i=5 2020-03-28T06:20:16.541 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.540 7fa0d766a840 20 remove_watcher() i=6 2020-03-28T06:20:16.591 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.590 7fa0d766a840 20 remove_watcher() i=7 2020-03-28T06:20:16.598 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T06:20:16.598 DEBUG:tasks.util.rgw: json result: [] 2020-03-28T06:20:16.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:20:16.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:20:16.599 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-03-28T06:20:16.826 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.825 7fe090ff9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:20:16.826 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.825 7fe083fff700 20 reqs_thread_entry: start 2020-03-28T06:20:16.826 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.825 7fe0c738f840 20 rados->read ofs=0 len=0 2020-03-28T06:20:16.837 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.837 7fe0c738f840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:16.837 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.837 7fe0c738f840 20 rados->read ofs=0 len=0 2020-03-28T06:20:16.848 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.847 7fe0c738f840 20 rados_obj.operate() r=0 bl.length=805 2020-03-28T06:20:16.848 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.847 7fe0c738f840 20 rados->read ofs=0 len=0 2020-03-28T06:20:16.853 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.852 7fe0c738f840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:16.853 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.852 7fe0c738f840 20 rados->read ofs=0 len=0 2020-03-28T06:20:16.860 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.859 7fe0c738f840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:16.860 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.859 7fe0c738f840 20 rados->read ofs=0 len=0 2020-03-28T06:20:16.867 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:16.866 7fe0c738f840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:20:16.941 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"55949d59-3b4c-4a71-a0a4-d4f6bb17ae9e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":""} 2020-03-28T06:20:16.942 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'metadata_heap': u'', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'user_email_pool': u'default.rgw.meta:users.email', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'placement_pools': [{u'val': {u'storage_classes': {u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'realm_id': u'', u'id': u'55949d59-3b4c-4a71-a0a4-d4f6bb17ae9e', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2020-03-28T06:20:16.942 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-03-28T06:20:16.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-03-28T06:20:16.942 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-03-28T06:20:17.110 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.110 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.131 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.131 7f9477dd4840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:17.132 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.131 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.141 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.141 7f9477dd4840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:17.141 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.141 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.148 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.147 7f9477dd4840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:17.148 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.147 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.157 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.156 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:17.363 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.362 7f9477dd4840 20 RGWRados::pool_iterate: got zonegroup_info.3691598c-2cf3-4c08-86e3-9c56f6b42b9a 2020-03-28T06:20:17.363 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.362 7f9477dd4840 20 RGWRados::pool_iterate: got zone_info.55949d59-3b4c-4a71-a0a4-d4f6bb17ae9e 2020-03-28T06:20:17.363 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.362 7f9477dd4840 20 RGWRados::pool_iterate: got zone_names.default 2020-03-28T06:20:17.363 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.362 7f9477dd4840 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-03-28T06:20:17.363 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.362 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.370 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.370 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:17.370 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.370 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.377 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=821 2020-03-28T06:20:17.378 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.377 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.385 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.385 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:17.386 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.385 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.393 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.392 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:20:17.393 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.392 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.397 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.396 7f9477dd4840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:17.397 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.396 7f9477dd4840 10 cannot find current period zonegroup using local zonegroup 2020-03-28T06:20:17.397 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.396 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.401 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.401 7f9477dd4840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:17.402 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.401 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.407 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.407 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:17.408 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.407 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.417 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.416 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:20:17.417 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.416 7f9477dd4840 20 zonegroup default 2020-03-28T06:20:17.417 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.416 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.420 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.420 7f9477dd4840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:17.421 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.420 7f9477dd4840 10 Cannot find current period zone using local zone 2020-03-28T06:20:17.421 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.420 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.425 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.424 7f9477dd4840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:17.425 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.424 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.432 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.431 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:17.432 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.431 7f9477dd4840 20 rados->read ofs=0 len=0 2020-03-28T06:20:17.439 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.439 7f9477dd4840 20 rados_obj.operate() r=0 bl.length=821 2020-03-28T06:20:17.440 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.439 7f9477dd4840 20 zone default 2020-03-28T06:20:17.745 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.744 7f9477dd4840 20 add_watcher() i=0 2020-03-28T06:20:17.745 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.744 7f9477dd4840 20 add_watcher() i=1 2020-03-28T06:20:17.745 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.744 7f9477dd4840 20 add_watcher() i=2 2020-03-28T06:20:17.745 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.744 7f9477dd4840 20 add_watcher() i=3 2020-03-28T06:20:17.746 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.744 7f9477dd4840 20 add_watcher() i=4 2020-03-28T06:20:17.746 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.744 7f9477dd4840 20 add_watcher() i=5 2020-03-28T06:20:17.746 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.744 7f9477dd4840 20 add_watcher() i=6 2020-03-28T06:20:17.770 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.770 7f9477dd4840 20 add_watcher() i=7 2020-03-28T06:20:17.770 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.770 7f9477dd4840 2 all 8 watchers are set, enabling cache 2020-03-28T06:20:17.839 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.838 7f9440ff9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:20:17.857 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.857 7f94237de700 20 reqs_thread_entry: start 2020-03-28T06:20:17.858 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.857 7f94217da700 20 reqs_thread_entry: start 2020-03-28T06:20:17.858 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.857 7f9477dd4840 20 init_complete bucket index max shards: 0 2020-03-28T06:20:17.858 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:17.857 7f939b7fe700 20 reqs_thread_entry: start 2020-03-28T06:20:18.093 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.092 7f9477dd4840 20 remove_watcher() i=0 2020-03-28T06:20:18.093 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.092 7f9477dd4840 2 removed watcher, disabling cache 2020-03-28T06:20:18.118 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.117 7f9477dd4840 20 remove_watcher() i=1 2020-03-28T06:20:18.145 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.145 7f9477dd4840 20 remove_watcher() i=2 2020-03-28T06:20:18.172 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.171 7f9477dd4840 20 remove_watcher() i=3 2020-03-28T06:20:18.193 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.193 7f9477dd4840 20 remove_watcher() i=4 2020-03-28T06:20:18.217 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.216 7f9477dd4840 20 remove_watcher() i=5 2020-03-28T06:20:18.241 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.240 7f9477dd4840 20 remove_watcher() i=6 2020-03-28T06:20:18.266 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.264 7f9477dd4840 20 remove_watcher() i=7 2020-03-28T06:20:18.272 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T06:20:18.272 DEBUG:tasks.util.rgw: json result: [] 2020-03-28T06:20:18.272 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:20:18.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-03-28T06:20:18.273 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-03-28T06:20:18.567 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.565 7f0d137fe700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-03-28T06:20:18.568 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.566 7f0d12ffd700 20 reqs_thread_entry: start 2020-03-28T06:20:18.568 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.566 7f0d4e79a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:18.579 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.577 7f0d4e79a840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:18.579 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.577 7f0d4e79a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:18.589 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.587 7f0d4e79a840 20 rados_obj.operate() r=0 bl.length=821 2020-03-28T06:20:18.589 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.587 7f0d4e79a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:18.593 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.592 7f0d4e79a840 20 rados_obj.operate() r=-2 bl.length=0 2020-03-28T06:20:18.593 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.592 7f0d4e79a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:18.601 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.599 7f0d4e79a840 20 rados_obj.operate() r=0 bl.length=46 2020-03-28T06:20:18.601 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.599 7f0d4e79a840 20 rados->read ofs=0 len=0 2020-03-28T06:20:18.608 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:18.606 7f0d4e79a840 20 rados_obj.operate() r=0 bl.length=348 2020-03-28T06:20:18.653 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"55949d59-3b4c-4a71-a0a4-d4f6bb17ae9e","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":""} 2020-03-28T06:20:18.653 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'metadata_heap': u'', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'user_email_pool': u'default.rgw.meta:users.email', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'placement_pools': [{u'val': {u'storage_classes': {u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'realm_id': u'', u'id': u'55949d59-3b4c-4a71-a0a4-d4f6bb17ae9e', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2020-03-28T06:20:18.653 INFO:tasks.rgw:Starting rgw... 2020-03-28T06:20:18.653 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2020-03-28T06:20:18.653 INFO:tasks.rgw:Using beast as radosgw frontend 2020-03-28T06:20:18.654 DEBUG:teuthology.misc:running c2.client.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c2.client.1.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--max-threads=1024'] 2020-03-28T06:20:18.654 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-03-28T06:20:18.654 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-03-28T06:20:18.659 INFO:tasks.rgw.c2.client.1:Started 2020-03-28T06:20:18.659 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2020-03-28T06:20:18.660 INFO:tasks.rgw:Using beast as radosgw frontend 2020-03-28T06:20:18.660 DEBUG:teuthology.misc:running c2.client.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c2.client.0.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--max-threads=1024'] 2020-03-28T06:20:18.660 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-03-28T06:20:18.660 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-03-28T06:20:18.663 INFO:tasks.rgw.c2.client.0:Started 2020-03-28T06:20:18.664 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000, 'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2020-03-28T06:20:18.664 INFO:tasks.rgw:Using beast as radosgw frontend 2020-03-28T06:20:18.664 DEBUG:teuthology.misc:running c1.client.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c1.client.0.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--max-threads=1024'] 2020-03-28T06:20:18.664 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-03-28T06:20:18.664 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-03-28T06:20:18.667 INFO:tasks.rgw.c1.client.0:Started 2020-03-28T06:20:18.667 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001, 'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2020-03-28T06:20:18.667 INFO:tasks.rgw:Using beast as radosgw frontend 2020-03-28T06:20:18.667 DEBUG:teuthology.misc:running c1.client.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/c1.client.1.log', '--time-stamp=yes', '--vgdb=yes', '--tool=memcheck', '--max-threads=1024'] 2020-03-28T06:20:18.668 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-03-28T06:20:18.668 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-03-28T06:20:18.670 INFO:tasks.rgw.c1.client.1:Started 2020-03-28T06:20:18.670 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi168.front.sepia.ceph.com:8001/ 2020-03-28T06:20:18.670 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001/ 2020-03-28T06:20:18.784 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:18.784 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:18.791 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:20:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:20:19.793 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001/ 2020-03-28T06:20:19.874 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:19.874 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:19.878 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:20:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:20:21.881 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001/ 2020-03-28T06:20:21.960 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:21.960 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:21.965 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:20:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:20:25.966 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001/ 2020-03-28T06:20:26.013 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:26.013 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:26.017 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:20:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:20:34.018 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001/ 2020-03-28T06:20:34.063 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:34.063 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:34.347 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 751 0 --:--:-- --:--:-- --:--:-- 750 2020-03-28T06:20:34.347 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-03-28T06:20:34.348 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi168.front.sepia.ceph.com:8000/ 2020-03-28T06:20:34.348 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000/ 2020-03-28T06:20:34.395 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:34.395 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:34.680 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 748 0 --:--:-- --:--:-- --:--:-- 750 2020-03-28T06:20:34.680 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-03-28T06:20:34.681 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi156.front.sepia.ceph.com:8000/ 2020-03-28T06:20:34.681 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000/ 2020-03-28T06:20:34.735 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:34.735 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:35.013 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 771 0 --:--:-- --:--:-- --:--:-- 772 2020-03-28T06:20:35.013 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2020-03-28T06:20:35.013 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi156.front.sepia.ceph.com:8001/ 2020-03-28T06:20:35.014 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001/ 2020-03-28T06:20:35.056 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:20:35.056 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:20:35.337 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 763 0 --:--:-- --:--:-- --:--:-- 761 2020-03-28T06:20:35.338 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2020-03-28T06:20:35.338 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-03-28T06:20:35.834 INFO:tasks.rgw_multisite:creating realm.. 2020-03-28T06:20:35.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:35.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:35.835 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:20:36.444 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","name":"test-realm","current_period":"3dbe3bbe-d45a-45ce-8b98-c958546e93a9","epoch":1} 2020-03-28T06:20:36.445 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-03-28T06:20:36.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:36.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:36.446 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:20:36.686 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:20:36.685 7f1ab6519840 -1 old format 2020-03-28T06:20:36.923 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T06:20:36.923 INFO:tasks.rgw_multisite:creating master zone.. 2020-03-28T06:20:36.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:36.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:36.924 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --endpoints http://smithi156.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:20:37.834 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T06:20:37.834 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-03-28T06:20:39.558 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-03-28T06:20:39.571 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-03-28T06:20:40.566 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-03-28T06:20:40.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:40.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:40.583 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:20:42.425 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":""} 2020-03-28T06:20:42.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:42.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:20:42.426 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:21:02.063 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"3894ea80-31ba-4896-8e09-e661b246b356","epoch":1,"predecessor_uuid":"3dbe3bbe-d45a-45ce-8b98-c958546e93a9","sync_status":[],"period_map":{"id":"3894ea80-31ba-4896-8e09-e661b246b356","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":2} 2020-03-28T06:21:02.064 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-03-28T06:21:02.064 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T06:21:02.080 INFO:tasks.rgw.c1.client.0.smithi156.stdout:2020-03-28 06:21:02.068 1b85f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 11449) UID: 0 2020-03-28T06:21:02.081 INFO:tasks.rgw.c1.client.0.smithi156.stdout:2020-03-28 06:21:02.072 4049040 -1 shutting down 2020-03-28T06:21:08.166 INFO:tasks.rgw.c1.client.0:Stopped 2020-03-28T06:21:08.167 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-03-28T06:21:08.167 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-03-28T06:21:08.171 INFO:tasks.rgw.c1.client.0:Started 2020-03-28T06:21:08.171 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000 2020-03-28T06:21:08.262 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:21:08.262 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:21:08.266 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:21:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:21:09.267 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000 2020-03-28T06:21:09.343 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:21:09.344 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:21:09.348 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:21:09.348 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:21:11.349 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000 2020-03-28T06:21:11.432 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:21:11.432 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:21:11.437 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:21:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:21:15.438 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000 2020-03-28T06:21:15.506 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:21:15.509 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:21:15.511 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:21:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:21:23.513 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8000 2020-03-28T06:21:23.559 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:21:23.559 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:21:23.840 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 757 0 --:--:-- --:--:-- --:--:-- 758 2020-03-28T06:21:23.841 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2020-03-28T06:21:23.842 INFO:tasks.rgw_multisite:creating admin user.. 2020-03-28T06:21:23.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:23.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:23.842 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:21:25.075 INFO:teuthology.orchestra.run.smithi156.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-03-28T06:21:25.353 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-03-28T06:21:25.353 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi156.front.sepia.ceph.com:8000', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:25.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi156.front.sepia.ceph.com:8000', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:25.354 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi156.front.sepia.ceph.com:8000 --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:21:26.476 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","name":"test-realm","current_period":"3894ea80-31ba-4896-8e09-e661b246b356","epoch":2} 2020-03-28T06:21:26.477 INFO:tasks.rgw_multisite:creating zone.. 2020-03-28T06:21:26.477 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:26.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:26.477 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --endpoints http://smithi168.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:21:27.113 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:21:27.117 7f83f2c52840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T06:21:27.113 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:21:27.117 7f83f2c52840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T06:21:27.193 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T06:21:27.194 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-03-28T06:21:28.552 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-03-28T06:21:28.566 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-03-28T06:21:29.562 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-03-28T06:21:29.575 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:29.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:29.575 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:21:31.630 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":""} 2020-03-28T06:21:31.631 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:31.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:21:31.631 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:21:51.764 INFO:teuthology.orchestra.run.smithi168.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T06:21:54.485 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"3894ea80-31ba-4896-8e09-e661b246b356","epoch":2,"predecessor_uuid":"3dbe3bbe-d45a-45ce-8b98-c958546e93a9","sync_status":[],"period_map":{"id":"3894ea80-31ba-4896-8e09-e661b246b356","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":2} 2020-03-28T06:21:54.486 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-03-28T06:21:54.486 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T06:21:54.510 INFO:tasks.rgw.c2.client.0.smithi168.stdout:2020-03-28 06:21:54.495 1b85f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 11534) UID: 0 2020-03-28T06:21:54.515 INFO:tasks.rgw.c2.client.0.smithi168.stdout:2020-03-28 06:21:54.504 4049040 -1 shutting down 2020-03-28T06:22:00.587 INFO:tasks.rgw.c2.client.0:Stopped 2020-03-28T06:22:00.588 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-03-28T06:22:00.588 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.0.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-03-28T06:22:00.592 INFO:tasks.rgw.c2.client.0:Started 2020-03-28T06:22:00.593 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000 2020-03-28T06:22:00.689 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:22:00.690 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:22:00.693 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:22:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:22:01.695 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000 2020-03-28T06:22:01.784 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:22:01.784 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:22:01.787 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:22:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:22:03.789 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000 2020-03-28T06:22:03.874 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:22:03.878 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:22:03.879 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:22:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:22:07.880 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000 2020-03-28T06:22:07.954 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:22:07.954 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:22:07.957 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8000; Connection refused 2020-03-28T06:22:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:22:15.959 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000 2020-03-28T06:22:16.027 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:22:16.027 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:22:16.126 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 2155 0 --:--:-- --:--:-- --:--:-- 2140 2020-03-28T06:22:16.127 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-03-28T06:22:16.127 INFO:tasks.rgw_multisite:creating zone.. 2020-03-28T06:22:16.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:22:16.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi156.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:22:16.128 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --endpoints http://smithi156.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:22:16.811 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:22:16.812 7f0be1033840 0 failed reading obj info from .rgw.root:zone_info.2c0a3e73-a280-4ccf-960d-d2c86331b6ff: (2) No such file or directory 2020-03-28T06:22:16.811 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 06:22:16.812 7f0be1033840 0 WARNING: could not read zone params for zone id=2c0a3e73-a280-4ccf-960d-d2c86331b6ff name=test-zone2 2020-03-28T06:22:16.865 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T06:22:16.865 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-03-28T06:22:17.814 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-03-28T06:22:17.828 INFO:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-03-28T06:22:18.938 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-03-28T06:22:18.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:22:18.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:22:18.959 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:22:23.676 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":""} 2020-03-28T06:22:23.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:22:23.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:22:23.677 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:22:49.437 INFO:teuthology.orchestra.run.smithi156.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T06:22:52.852 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"3894ea80-31ba-4896-8e09-e661b246b356","epoch":3,"predecessor_uuid":"3dbe3bbe-d45a-45ce-8b98-c958546e93a9","sync_status":[],"period_map":{"id":"3894ea80-31ba-4896-8e09-e661b246b356","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":2} 2020-03-28T06:22:52.853 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-03-28T06:22:52.853 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T06:22:52.874 INFO:tasks.rgw.c1.client.1.smithi156.stdout:2020-03-28 06:22:52.859 1b85f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 11447) UID: 0 2020-03-28T06:22:52.875 INFO:tasks.rgw.c1.client.1.smithi156.stdout:2020-03-28 06:22:52.863 4049040 -1 shutting down 2020-03-28T06:22:58.954 INFO:tasks.rgw.c1.client.1:Stopped 2020-03-28T06:22:58.954 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-03-28T06:22:58.955 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-03-28T06:22:58.957 INFO:tasks.rgw.c1.client.1:Started 2020-03-28T06:22:58.957 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T06:22:59.008 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:22:59.009 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:22:59.012 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:22:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:23:00.013 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T06:23:00.060 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:23:00.060 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:23:00.064 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:23:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:23:02.065 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T06:23:02.109 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:23:02.109 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:23:02.113 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:23:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:23:06.114 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T06:23:06.191 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:23:06.191 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:23:06.193 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:23:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:23:14.194 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T06:23:14.247 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:23:14.247 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:23:14.442 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 1096 0 --:--:-- --:--:-- --:--:-- 1097 2020-03-28T06:23:14.443 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2020-03-28T06:23:14.443 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-03-28T06:23:14.443 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi156.front.sepia.ceph.com:8000', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--url', 'http://smithi156.front.sepia.ceph.com:8000', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:14.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi156.front.sepia.ceph.com:8000', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--url', 'http://smithi156.front.sepia.ceph.com:8000', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:14.444 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi156.front.sepia.ceph.com:8000 --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --url http://smithi156.front.sepia.ceph.com:8000 --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:23:15.170 INFO:tasks.rgw.c2.client.0.smithi168.stdout:2020-03-28 06:23:15.166 148888700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.b24b1e69-ff48-4356-b1b9-8937c2aaecdc.control err=-107 2020-03-28T06:23:15.389 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","name":"test-realm","current_period":"3894ea80-31ba-4896-8e09-e661b246b356","epoch":2} 2020-03-28T06:23:15.389 INFO:tasks.rgw_multisite:creating zone.. 2020-03-28T06:23:15.390 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', u'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:15.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'BQ8OGVQAZXH9K7HK', '--secret', 'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', u'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:15.390 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --endpoints http://smithi168.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:23:16.011 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:23:16.017 7f3af7dd0840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T06:23:16.012 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:23:16.017 7f3af7dd0840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T06:23:16.016 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:23:16.022 7f3af7dd0840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T06:23:16.016 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 06:23:16.022 7f3af7dd0840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T06:23:16.054 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T06:23:16.054 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-03-28T06:23:16.917 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-03-28T06:23:16.935 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-03-28T06:23:18.062 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-03-28T06:23:18.073 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:18.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:18.074 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:23:20.040 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2020-03-28T06:23:20.040 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:20.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:23:20.041 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T06:23:47.112 INFO:teuthology.orchestra.run.smithi168.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T06:23:49.550 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"3894ea80-31ba-4896-8e09-e661b246b356","epoch":4,"predecessor_uuid":"3dbe3bbe-d45a-45ce-8b98-c958546e93a9","sync_status":[],"period_map":{"id":"3894ea80-31ba-4896-8e09-e661b246b356","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":2} 2020-03-28T06:23:49.551 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-03-28T06:23:49.551 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T06:23:49.566 INFO:tasks.rgw.c2.client.1.smithi168.stdout:2020-03-28 06:23:49.561 1b85f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 11532) UID: 0 2020-03-28T06:23:49.574 INFO:tasks.rgw.c2.client.1.smithi168.stdout:2020-03-28 06:23:49.564 4049040 -1 shutting down 2020-03-28T06:23:55.653 INFO:tasks.rgw.c2.client.1:Stopped 2020-03-28T06:23:55.653 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-03-28T06:23:55.654 INFO:teuthology.orchestra.run.smithi168:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-03-28T06:23:55.668 INFO:tasks.rgw.c2.client.1:Started 2020-03-28T06:23:55.668 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-03-28T06:23:55.719 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:23:55.719 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:23:55.728 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:23:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:23:56.729 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-03-28T06:23:56.774 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:23:56.774 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:23:56.778 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:23:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:23:58.780 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-03-28T06:23:58.827 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:23:58.827 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:23:58.829 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:23:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:24:02.831 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-03-28T06:24:02.912 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:24:02.912 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:24:02.912 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi168.front.sepia.ceph.com:8001; Connection refused 2020-03-28T06:24:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T06:24:10.913 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-03-28T06:24:10.961 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T06:24:10.962 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-03-28T06:24:11.080 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 1120 0 --:--:-- --:--:-- --:--:-- 1116 100 134 100 134 0 0 1118 0 --:--:-- --:--:-- --:--:-- 1116 2020-03-28T06:24:11.081 INFO:teuthology.orchestra.run.smithi168.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005e7eed8b-1116-test-zone4","HostId":"1116-test-zone4-test-zonegroup"} 2020-03-28T06:24:11.082 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-03-28T06:24:11.082 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-03-28T06:24:11.123 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-03-28T06:24:11.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:24:11.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T06:24:11.124 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T06:24:12.555 INFO:teuthology.orchestra.run.smithi156.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"HPX470M26LNLIX5T3HRY","secret_key":"SHY0LfAbaF6cxBaNuuxMttRKJ96mhE7IVA1nYwUu"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-03-28T06:24:12.777 INFO:rgw_multi.tests:meta checkpoint 2020-03-28T06:24:12.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:12.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:12.778 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:13.830 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:24:14.098 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376652.440666_46.1","last_update":"2020-03-28 06:24:12.440666Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:24:14.099 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:24:14.099 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:14.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:14.099 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:15.541 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:15.542 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:24:20.542 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:20.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:20.543 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:21.884 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:21.885 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:24:26.886 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:26.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:26.886 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:28.205 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:28.206 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:24:28.206 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:24:28.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:28.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:28.207 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:29.540 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:29.541 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:24:34.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:34.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:34.542 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:36.026 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:36.026 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:24:41.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:41.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:41.028 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:42.521 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:42.522 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:24:47.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:47.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:47.524 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:48.945 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:48.945 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:24:53.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:53.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:24:53.948 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:24:55.409 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:24:55.409 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:25:00.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:00.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:00.411 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:01.925 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:01.925 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:25:06.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:06.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:06.926 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:08.326 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:08.327 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:25:13.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:13.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:13.328 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:14.663 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:14.663 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:25:19.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:19.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:19.665 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:21.116 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:21.117 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1585376652.440666_46.1 target= 2020-03-28T06:25:26.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:26.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:26.119 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:27.523 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:27.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:25:27.523 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:25:27.524 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:27.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:27.524 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:28.956 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:28.956 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:25:28.957 INFO:rgw_multi.tests:meta checkpoint 2020-03-28T06:25:28.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:28.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:28.957 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:30.012 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:25:30.282 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376652.440666_46.1","last_update":"2020-03-28 06:24:12.440666Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:25:30.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:25:30.282 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:30.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:30.283 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:31.579 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:31.579 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:25:31.579 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:25:31.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:31.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:31.580 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:33.011 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:33.012 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:25:33.012 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:25:33.012 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:33.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:33.013 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:34.593 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:34.594 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:25:34.594 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-03-28T06:25:34.599 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-1 2020-03-28T06:25:44.180 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-2 2020-03-28T06:25:55.816 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-3 2020-03-28T06:25:56.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:56.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:56.831 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:57.924 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:25:58.170 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376652.440666_46.1","last_update":"2020-03-28 06:24:12.440666Z"},{"marker":"1_1585376756.492951_88.1","last_update":"2020-03-28 06:25:56.492951Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:25:58.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:25:58.170 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:58.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:58.171 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:25:59.382 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376756.492951_88.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:25:56.492951Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:25:59.382 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:25:59.383 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:25:59.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:59.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:25:59.383 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:00.652 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376756.492951_88.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:25:56.492951Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:00.653 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:26:00.653 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:26:00.653 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:00.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:00.653 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:02.099 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376756.492951_88.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:25:56.492951Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:02.099 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:26:03.221 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-03-28T06:26:03.221 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-4 2020-03-28T06:26:03.615 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-5 2020-03-28T06:26:05.088 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-6 2020-03-28T06:26:06.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:06.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:06.133 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:07.093 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:26:07.336 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376652.440666_46.1","last_update":"2020-03-28 06:24:12.440666Z"},{"marker":"1_1585376765.686952_100.1","last_update":"2020-03-28 06:26:05.686952Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:26:07.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:26:07.336 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:07.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:07.337 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:08.859 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376765.686952_100.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:05.686952Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:08.859 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:26:08.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:26:08.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:08.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:08.860 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:10.213 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376765.686952_100.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:05.686952Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:10.213 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:26:10.213 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:26:10.214 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:10.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:10.214 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:11.666 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376765.686952_100.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:05.686952Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:11.666 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:26:21.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:21.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:21.867 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:22.918 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:26:23.322 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376652.440666_46.1","last_update":"2020-03-28 06:24:12.440666Z"},{"marker":"1_1585376780.715973_154.1","last_update":"2020-03-28 06:26:20.715973Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:26:23.322 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:26:23.322 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:23.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:23.323 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:24.689 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376780.715973_154.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:20.715973Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:24.690 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:26:24.690 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:26:24.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:24.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:24.691 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:25.263 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:08:57.240 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:26:25.314 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:08:57.290 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:26:25.768 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:08:57.744 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:26:26.358 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376780.715973_154.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:20.715973Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:26.359 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:26:26.359 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:26:26.359 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:26.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:26.360 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:27.873 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376780.715973_154.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:20.715973Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:27.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:26:28.654 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-03-28T06:26:28.655 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-7 2020-03-28T06:26:29.116 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-8 2020-03-28T06:26:30.281 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-9 2020-03-28T06:26:30.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:30.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:30.913 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:31.949 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:26:32.245 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376652.440666_46.1","last_update":"2020-03-28 06:24:12.440666Z"},{"marker":"1_1585376790.668987_167.1","last_update":"2020-03-28 06:26:30.668987Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:26:32.246 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:26:32.246 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:32.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:32.247 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:33.571 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376790.668987_167.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:30.668987Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:33.572 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:26:33.572 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:26:33.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:33.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:33.573 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:34.970 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376790.668987_167.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:30.668987Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:34.970 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:26:34.970 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:26:34.970 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:34.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:34.971 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:36.319 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376790.668987_167.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:30.668987Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:36.320 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:26:38.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:38.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:38.715 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:39.710 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:26:39.980 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376652.440666_46.1","last_update":"2020-03-28 06:24:12.440666Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:26:39.980 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:26:39.981 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:39.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:39.981 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:41.355 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:41.356 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:26:41.356 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:26:41.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:41.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:41.357 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:42.763 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:42.764 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:26:42.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:26:42.764 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:42.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:42.765 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:44.358 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376652.440666_46.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:24:12.440666Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:44.358 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:26:45.058 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-03-28T06:26:45.059 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-10 2020-03-28T06:26:45.446 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-11 2020-03-28T06:26:46.439 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-12 2020-03-28T06:26:53.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:53.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:53.195 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:54.865 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:26:55.785 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585376807.210517_58.1","last_update":"2020-03-28 06:26:47.210517Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:26:55.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:26:55.786 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:55.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:55.787 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:57.052 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376807.210517_58.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:26:47.210517Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:57.053 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:26:57.053 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:26:57.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:57.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:57.054 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:26:58.832 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376807.210517_58.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:26:47.210517Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:26:58.832 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:26:58.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:26:58.833 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:58.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:26:58.833 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:00.135 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585376807.210517_58.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:26:47.210517Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:27:00.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:27:00.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:00.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:00.137 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:01.466 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T06:27:01.466 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-10 2020-03-28T06:27:01.466 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:01.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:01.467 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:03.041 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.8.5"}}}] 2020-03-28T06:27:03.042 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-10 2020-03-28T06:27:03.042 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-10 zones={test-zone2, test-zone1} 2020-03-28T06:27:03.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:27:03.290 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:03.290 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:03.291 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:03.291 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:03.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:27:03.408 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:03.409 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:03.409 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:03.409 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:03.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:03.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:04.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:04.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:05.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:05.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:05.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:06.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:06.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:06.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:07.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:07.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:08.080 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-10 zones={test-zone2, test-zone1} 2020-03-28T06:27:08.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:08.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:08.081 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:09.259 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T06:27:09.259 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-10 2020-03-28T06:27:09.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:09.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:09.260 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:10.523 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.8.5"}}}] 2020-03-28T06:27:10.523 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-10 2020-03-28T06:27:10.524 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-10 zones={test-zone3, test-zone1} 2020-03-28T06:27:10.612 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:27:10.674 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:10.674 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:10.675 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:10.675 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:10.675 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:27:10.731 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:10.731 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:10.731 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:10.731 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:10.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:11.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:11.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:11.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:11.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:12.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:12.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:12.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:12.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:13.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:13.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:13.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:13.836 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-10 zones={test-zone3, test-zone1} 2020-03-28T06:27:13.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:13.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:13.837 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:15.151 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T06:27:15.151 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-10 2020-03-28T06:27:15.151 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:15.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:15.152 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:16.915 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000008.8.5"}}}] 2020-03-28T06:27:16.916 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-10 2020-03-28T06:27:16.916 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:16.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:16.917 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:18.304 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.9.5"}]} 2020-03-28T06:27:18.304 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-11 2020-03-28T06:27:18.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:18.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:18.305 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:19.780 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.9.5"}}}] 2020-03-28T06:27:19.781 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-11 2020-03-28T06:27:19.781 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-11 zones={test-zone1, test-zone2} 2020-03-28T06:27:19.900 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:27:20.012 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:20.012 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:20.012 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:20.012 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:20.012 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:27:20.095 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:20.096 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:20.096 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:20.096 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:20.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:20.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:20.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:21.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:21.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:22.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:22.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:22.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:23.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:23.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:23.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:24.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:24.876 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-11 zones={test-zone1, test-zone2} 2020-03-28T06:27:24.876 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:24.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:24.877 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:26.137 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.9.5"}]} 2020-03-28T06:27:26.137 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-11 2020-03-28T06:27:26.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:26.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:26.138 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:27.612 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.9.5"}}}] 2020-03-28T06:27:27.612 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-11 2020-03-28T06:27:27.613 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-11 zones={test-zone3, test-zone2} 2020-03-28T06:27:27.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:27:27.831 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:27.831 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:27.831 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:27.832 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:27.832 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:27:27.922 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-03-28T06:27:27.922 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-03-28T06:27:27.922 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-03-28T06:27:27.923 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:27:28.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:28.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:28.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-03-28T06:27:29.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:29.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:29.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-03-28T06:27:29.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:29.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:30.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-03-28T06:27:30.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:30.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:31.164 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:08:57.200 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:27:31.241 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:08:57.278 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:27:31.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:27:31.472 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-11 zones={test-zone3, test-zone2} 2020-03-28T06:27:31.473 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:31.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:31.473 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:31.806 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:08:57.843 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:27:33.082 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.9.5"}]} 2020-03-28T06:27:33.083 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-11 2020-03-28T06:27:33.083 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:33.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:33.084 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:34.387 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:27:34.388 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:27:39.389 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:39.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:39.389 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:41.245 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:27:41.245 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:27:46.246 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:46.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:46.247 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:47.713 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:27:47.714 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:27:52.715 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:52.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:52.715 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:27:54.214 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:27:54.214 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:27:59.215 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:59.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:27:59.216 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:00.621 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:00.622 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:05.623 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:05.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:05.624 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:07.096 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:07.097 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:12.097 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:12.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:12.098 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:13.517 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:13.518 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:18.519 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:18.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:18.520 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:20.216 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:20.217 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:25.218 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:25.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:25.218 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:26.664 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:26.664 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:31.665 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:31.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:31.666 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:33.093 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:33.094 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:38.095 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:38.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:38.095 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:39.601 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:39.601 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:44.602 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:44.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:44.602 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:46.192 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:46.193 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:51.194 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:51.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:51.194 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:52.600 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:52.600 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:28:57.601 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:57.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:28:57.602 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:28:59.168 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:28:59.169 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:04.170 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:04.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:04.171 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:05.567 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:05.568 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:10.568 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:10.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:10.569 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:12.234 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:12.234 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:17.235 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:17.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:17.236 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:18.750 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:18.750 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:23.752 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:23.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:23.752 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:25.083 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:25.084 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:30.084 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:30.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:30.085 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:31.523 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:31.524 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:36.524 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:36.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:36.525 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:38.040 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:38.040 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:43.041 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:43.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:43.042 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:44.440 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:44.441 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:49.441 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:49.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:49.442 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:51.154 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:51.155 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:29:56.155 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:56.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:29:56.156 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:29:57.632 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:29:57.633 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:02.633 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:02.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:02.634 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:04.164 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:04.164 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:09.165 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:09.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:09.166 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:10.563 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:10.563 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:15.563 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:15.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:15.564 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:17.177 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:17.178 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:22.179 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:22.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:22.180 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:23.953 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:23.953 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:28.954 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:28.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:28.954 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:30.420 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:30.420 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:35.421 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:35.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:35.422 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:36.990 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:36.991 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:41.992 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:41.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:41.993 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:43.587 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:43.587 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:48.588 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:48.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:48.589 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:50.037 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:50.037 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:30:55.038 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:55.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:30:55.039 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:30:56.473 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:30:56.473 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:01.474 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:01.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:01.475 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:03.275 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:03.276 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:08.277 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:08.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:08.277 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:09.916 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:09.917 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:14.917 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:14.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:14.918 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:16.353 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:16.354 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:21.355 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:21.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:21.355 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:22.721 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:22.721 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:27.722 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:27.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:27.723 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:29.194 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:29.194 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:34.195 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:34.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:34.196 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:35.897 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:35.897 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:40.898 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:40.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:40.899 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:42.500 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:42.500 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:47.501 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:47.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:47.502 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:49.062 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:49.062 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:31:54.064 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:54.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:31:54.064 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:31:55.792 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:31:55.792 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:00.793 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:00.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:00.794 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:02.347 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:02.348 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:07.350 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:07.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:07.351 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:08.821 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:08.822 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:13.823 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:13.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:13.824 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:15.319 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:15.319 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:20.319 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:20.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:20.320 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:21.810 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:21.811 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:26.811 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:26.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:26.812 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:28.377 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:28.377 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:33.379 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:33.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:33.379 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:34.933 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:34.933 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:39.934 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:39.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:39.935 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:41.472 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:41.472 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:46.473 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:46.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:46.474 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:47.951 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:47.951 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:52.953 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:52.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:52.953 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:32:54.324 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:32:54.324 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:32:59.326 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:59.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:32:59.326 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:00.946 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:00.946 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:05.948 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:05.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:05.948 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:07.463 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:07.463 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:12.464 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:12.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:12.465 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:14.055 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:14.055 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:19.056 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:19.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:19.057 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:20.677 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:20.677 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:25.679 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:25.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:25.679 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:27.321 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:27.322 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:32.323 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:32.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:32.324 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:33.972 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:33.972 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:38.973 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:38.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:38.973 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:40.452 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:40.453 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:45.453 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:45.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:45.454 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:47.031 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:47.031 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:52.031 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:52.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:52.032 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:33:53.597 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:33:53.598 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:33:58.598 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:58.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:33:58.599 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:00.204 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:34:00.205 WARNING:rgw_multi.tests:bucket axxsjh-11 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.9.5 target= 2020-03-28T06:34:05.222 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... FAIL 2020-03-28T06:34:05.222 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-13 2020-03-28T06:34:05.608 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-14 2020-03-28T06:34:06.293 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-15 2020-03-28T06:34:08.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:08.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:08.677 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:10.252 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:34:10.826 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585377247.211607_78.1","last_update":"2020-03-28 06:34:07.211607Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:34:10.827 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:34:10.827 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:10.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:10.828 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:13.061 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585377247.211607_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:34:07.211607Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:34:13.061 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:34:13.061 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:34:13.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:13.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:13.062 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:14.257 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585377247.211607_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:34:07.211607Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:34:14.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:34:14.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:34:14.258 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:14.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:14.258 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:15.795 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585377247.211607_78.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:34:07.211607Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:34:15.795 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:34:15.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:15.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:15.796 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:17.041 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000002.2.5"}]} 2020-03-28T06:34:17.042 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-13 2020-03-28T06:34:17.042 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:17.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:17.042 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:18.580 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.2.5"}}}] 2020-03-28T06:34:18.580 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-13 2020-03-28T06:34:18.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-13 zones={test-zone2, test-zone1} 2020-03-28T06:34:18.707 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:34:18.803 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:18.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:34:18.898 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:19.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:19.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:19.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:20.124 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-13 zones={test-zone2, test-zone1} 2020-03-28T06:34:20.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:20.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:20.125 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:21.519 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000002.2.5"}]} 2020-03-28T06:34:21.519 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-13 2020-03-28T06:34:21.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:21.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:21.520 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:22.784 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.2.5"}}}] 2020-03-28T06:34:22.785 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-13 2020-03-28T06:34:22.785 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-13 zones={test-zone3, test-zone1} 2020-03-28T06:34:22.893 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:34:22.942 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:22.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:34:22.997 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:23.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:23.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:23.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:23.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-13 zones={test-zone3, test-zone1} 2020-03-28T06:34:23.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:23.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:23.982 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:25.230 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000002.2.5"}]} 2020-03-28T06:34:25.230 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-13 2020-03-28T06:34:25.231 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:25.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:25.231 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:26.832 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.2.5"}}}] 2020-03-28T06:34:26.832 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-13 2020-03-28T06:34:26.832 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:26.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:26.833 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:28.054 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.3.5"}]} 2020-03-28T06:34:28.054 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-14 2020-03-28T06:34:28.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:28.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:28.055 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:29.434 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.3.5"}}}] 2020-03-28T06:34:29.434 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-14 2020-03-28T06:34:29.434 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-14 zones={test-zone1, test-zone2} 2020-03-28T06:34:29.560 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:34:29.605 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:29.605 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:34:29.688 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:29.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:30.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:30.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:30.815 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-14 zones={test-zone1, test-zone2} 2020-03-28T06:34:30.816 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:30.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:30.816 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:32.095 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.3.5"}]} 2020-03-28T06:34:32.095 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-14 2020-03-28T06:34:32.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:32.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:32.096 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:33.656 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.3.5"}}}] 2020-03-28T06:34:33.657 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-14 2020-03-28T06:34:33.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-14 zones={test-zone3, test-zone2} 2020-03-28T06:34:33.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T06:34:33.849 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:33.849 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T06:34:33.933 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-03-28T06:34:34.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:34.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:34.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-03-28T06:34:35.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-14 zones={test-zone3, test-zone2} 2020-03-28T06:34:35.020 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:35.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:35.020 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:35.322 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:17:07.305 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:34:35.332 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:17:07.315 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:34:35.701 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:17:07.684 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:34:36.217 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.3.5"}]} 2020-03-28T06:34:36.217 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-14 2020-03-28T06:34:36.217 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:36.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:36.218 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:37.537 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:34:37.538 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:34:42.538 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:42.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:42.539 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:44.240 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:34:44.241 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:34:46.170 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:17:18.153 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:34:46.172 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:17:18.156 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:34:46.205 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:17:18.189 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:34:49.242 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:49.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:49.243 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:50.823 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:34:50.824 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:34:55.824 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:55.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:34:55.825 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:34:57.622 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:34:57.623 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:02.626 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:02.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:02.626 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:04.203 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:04.203 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:09.204 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:09.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:09.205 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:10.689 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:10.690 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:15.690 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:15.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:15.691 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:17.163 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:17.164 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:22.165 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:22.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:22.166 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:23.908 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:23.909 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:28.909 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:28.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:28.910 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:30.370 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:30.371 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:35.372 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:35.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:35.373 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:36.910 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:36.910 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:41.911 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:41.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:41.912 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:43.618 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:43.618 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:48.618 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:48.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:48.619 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:50.320 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:50.321 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:35:51.928 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:17:17.977 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:35:51.982 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:17:18.031 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:35:52.356 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:17:18.405 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:35:53.345 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:17:19.395 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:35:53.349 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:17:19.398 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:35:53.397 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:17:19.446 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:35:55.322 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:55.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:35:55.322 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:35:56.902 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:35:56.903 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:01.904 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:01.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:01.904 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:03.600 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:03.601 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:08.602 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:08.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:08.602 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:10.036 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:10.036 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:15.037 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:15.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:15.038 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:16.742 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:16.742 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:21.743 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:21.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:21.743 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:23.337 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:23.337 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:28.339 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:28.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:28.339 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:29.792 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:29.793 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:34.794 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:34.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:34.949 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:36.551 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:36.551 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:41.553 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:41.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:41.553 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:43.027 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:43.028 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:48.028 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:48.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:48.029 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:49.729 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:49.730 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:36:54.731 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:54.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:36:54.731 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:36:56.273 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:36:56.273 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:01.274 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:01.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:01.275 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:02.856 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:02.857 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:07.858 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:07.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:07.859 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:09.597 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:09.598 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:14.599 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:14.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:14.600 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:16.374 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:16.375 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:21.376 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:21.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:21.376 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:23.041 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:23.041 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:28.041 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:28.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:28.042 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:29.631 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:29.631 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:34.632 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:34.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:34.633 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:36.183 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:36.184 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:41.185 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:41.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:41.185 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:42.861 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:42.862 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:47.862 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:47.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:47.863 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:49.507 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:49.508 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:37:54.508 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:54.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:37:54.510 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:37:56.029 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:37:56.030 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:01.031 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:01.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:01.031 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:02.515 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:02.516 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:07.517 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:07.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:07.517 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:09.166 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:09.166 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:14.168 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:14.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:14.168 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:15.612 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:15.612 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:20.613 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:20.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:20.614 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:22.062 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:22.063 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:27.063 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:27.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:27.064 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:28.706 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:28.706 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:33.707 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:33.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:33.708 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:35.329 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:35.330 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:40.331 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:40.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:40.331 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:41.871 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:41.871 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:46.872 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:46.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:46.872 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:48.435 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:48.451 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:53.452 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:53.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:53.453 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:38:54.941 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:38:54.942 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:38:59.942 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:59.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:38:59.943 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:01.499 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:01.500 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:06.500 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:06.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:06.501 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:07.971 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:07.972 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:12.973 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:12.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:12.974 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:14.437 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:14.437 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:19.438 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:19.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:19.438 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:20.982 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:20.982 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:25.984 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:25.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:25.985 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:27.653 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:27.654 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:32.655 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:32.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:32.655 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:34.240 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:34.240 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:39.241 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:39.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:39.242 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:40.800 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:40.801 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:45.801 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:45.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:45.802 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:47.357 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:47.357 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:52.357 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:52.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:52.358 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:39:54.056 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:39:54.056 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:39:59.057 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:59.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:39:59.058 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:00.623 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:00.623 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:05.625 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:05.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:05.625 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:07.143 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:07.143 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:12.144 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:12.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:12.145 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:13.753 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:13.754 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:18.755 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:18.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:18.755 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:20.473 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:20.474 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:25.475 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:25.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:25.477 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:27.062 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:27.062 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:32.063 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:32.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:32.063 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:33.581 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:33.582 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:38.582 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:38.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:38.583 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:40.204 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:40.204 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:45.205 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:45.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:45.206 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:46.704 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:46.704 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:51.706 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:51.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:51.706 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:53.294 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:53.295 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:40:58.295 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:58.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:40:58.296 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:40:59.669 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:40:59.671 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:41:04.671 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:04.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:04.672 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:06.224 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:41:06.225 WARNING:rgw_multi.tests:bucket axxsjh-14 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.3.5 target= 2020-03-28T06:41:11.228 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... FAIL 2020-03-28T06:41:11.229 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-16 2020-03-28T06:41:11.733 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-17 2020-03-28T06:41:12.787 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-18 2020-03-28T06:41:15.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:15.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:15.239 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:16.181 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:41:16.471 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585377675.127242_96.1","last_update":"2020-03-28 06:41:15.127242Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:41:16.471 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:41:16.471 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:16.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:16.472 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:17.944 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585377675.127242_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:41:15.127242Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:41:17.945 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:41:17.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:41:17.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:17.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:17.946 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:19.342 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585377675.127242_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:41:15.127242Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:41:19.343 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:41:19.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:41:19.343 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:19.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:19.344 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:21.104 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585377675.127242_96.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:41:15.127242Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:41:21.105 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:41:21.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:21.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:21.847 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:23.287 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000003.4.10"}]} 2020-03-28T06:41:23.288 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-16 2020-03-28T06:41:23.289 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:23.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:23.289 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:24.843 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"GqJTBRTJoSxrSgBKvclrQX4KqjUqIFx","ns":""},"count":1},"inc_marker":{"position":"00000000004.6.9"}}}] 2020-03-28T06:41:24.843 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-16 2020-03-28T06:41:24.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:24.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:24.844 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:26.890 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000006.10.9"}]} 2020-03-28T06:41:26.891 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-16 2020-03-28T06:41:26.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:26.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:26.891 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:28.367 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"GqJTBRTJoSxrSgBKvclrQX4KqjUqIFx","ns":""},"count":1},"inc_marker":{"position":"00000000006.10.9"}}}] 2020-03-28T06:41:28.367 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-16 2020-03-28T06:41:28.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:28.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:28.368 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:29.652 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000007.12.9"}]} 2020-03-28T06:41:29.652 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-16 2020-03-28T06:41:29.652 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:29.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:29.653 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:29.895 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:24:01.869 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:41:29.897 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:24:01.872 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:41:30.204 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:24:02.179 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:41:31.480 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"GqJTBRTJoSxrSgBKvclrQX4KqjUqIFx","ns":""},"count":1},"inc_marker":{"position":"00000000007.12.9"}}}] 2020-03-28T06:41:31.480 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-16 2020-03-28T06:41:31.995 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:31.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:31.995 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:33.268 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000003.9.10"}]} 2020-03-28T06:41:33.268 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-17 2020-03-28T06:41:33.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:33.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:33.270 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:35.051 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"k7sdrZdvr3SJIHLS5fcx2wnijN0zquR","ns":""},"count":1},"inc_marker":{"position":"0#00000000003.9.10"}}}] 2020-03-28T06:41:35.051 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-17 2020-03-28T06:41:35.052 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:35.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:35.052 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:36.432 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000005.12.9"}]} 2020-03-28T06:41:36.433 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-17 2020-03-28T06:41:36.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:36.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:36.434 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:37.981 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"k7sdrZdvr3SJIHLS5fcx2wnijN0zquR","ns":""},"count":1},"inc_marker":{"position":"00000000005.12.9"}}}] 2020-03-28T06:41:37.982 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-17 2020-03-28T06:41:37.982 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:37.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:37.983 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:39.457 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000007.16.9"}]} 2020-03-28T06:41:39.457 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-17 2020-03-28T06:41:39.458 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:39.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:39.458 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:40.950 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:41:40.951 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:41:45.952 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:45.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:45.953 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:47.513 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:41:47.514 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:41:52.514 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:52.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:52.515 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:41:54.164 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:41:54.165 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:41:59.166 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:59.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:41:59.167 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:00.769 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:00.770 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:05.770 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:05.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:05.771 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:07.216 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:07.216 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:12.217 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:12.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:12.218 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:13.714 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:13.715 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:18.716 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:18.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:18.717 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:20.383 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:20.384 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:25.385 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:25.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:25.385 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:26.951 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:26.952 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:31.953 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:31.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:31.953 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:33.637 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:33.637 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:38.638 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:38.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:38.639 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:40.259 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:40.260 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:45.261 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:45.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:45.262 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:46.713 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:46.713 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:48.957 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:24:15.011 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:42:49.007 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:24:15.060 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:42:49.345 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:24:15.398 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:42:51.714 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:51.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:51.715 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:42:53.456 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:42:53.457 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:42:58.458 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:58.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:42:58.458 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:00.203 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:00.205 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:05.205 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:05.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:05.206 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:06.821 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:06.822 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:07.174 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:25:39.147 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:43:07.177 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:25:39.150 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:43:07.252 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:25:39.225 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:43:11.822 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:11.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:11.823 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:13.701 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:13.701 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:18.702 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:18.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:18.703 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:20.223 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:20.224 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:25.225 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:25.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:25.226 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:26.767 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:26.768 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:31.769 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:31.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:31.770 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:33.504 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:33.505 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:38.510 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:38.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:38.511 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:40.381 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:40.383 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:45.384 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:45.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:45.385 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:46.880 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:46.880 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:51.881 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:51.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:51.881 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:43:53.548 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:43:53.548 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:43:58.549 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:58.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:43:58.550 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:00.159 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:00.159 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:05.161 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:05.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:05.161 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:06.986 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:06.987 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:11.988 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:11.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:11.989 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:12.640 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:25:38.695 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:44:12.644 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:25:38.699 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:44:12.699 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:25:38.754 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:44:13.719 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:13.720 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:18.721 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:18.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:18.723 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:20.310 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:20.311 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:25.312 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:25.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:25.312 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:26.940 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:26.940 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:31.941 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:31.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:31.941 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:33.648 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:33.648 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:38.649 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:38.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:38.650 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:40.395 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:40.395 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:45.396 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:45.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:45.397 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:46.884 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:46.885 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:51.886 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:51.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:51.886 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:44:53.702 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:44:53.702 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:44:58.703 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:58.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:44:58.704 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:00.459 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:00.460 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:05.461 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:05.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:05.462 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:07.039 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:07.039 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:12.040 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:12.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:12.041 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:13.662 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:13.662 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:18.664 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:18.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:18.664 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:20.506 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:20.506 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:25.507 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:25.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:25.507 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:27.046 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:27.046 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:32.047 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:32.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:32.048 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:33.675 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:33.676 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:38.677 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:38.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:38.677 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:40.458 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:40.458 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:45.460 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:45.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:45.460 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:47.054 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:47.054 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:52.055 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:52.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:52.056 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:45:53.774 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:45:53.774 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:45:58.775 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:58.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:45:58.776 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:00.397 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:00.398 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:05.398 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:05.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:05.399 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:07.067 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:07.068 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:12.069 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:12.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:12.070 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:13.777 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:13.778 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:18.778 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:18.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:18.779 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:20.364 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:20.364 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:25.365 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:25.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:25.365 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:26.934 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:26.934 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:31.935 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:31.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:31.936 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:33.763 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:33.764 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:38.765 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:38.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:38.766 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:40.591 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:40.591 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:45.592 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:45.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:45.593 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:47.217 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:47.217 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:52.218 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:52.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:52.219 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:46:53.797 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:46:53.797 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:46:58.798 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:58.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:46:58.798 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:00.580 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:00.580 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:05.581 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:05.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:05.582 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:07.041 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:07.041 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:12.043 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:12.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:12.043 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:13.720 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:13.721 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:18.722 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:18.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:18.722 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:20.333 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:20.333 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:25.334 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:25.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:25.335 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:26.861 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:26.861 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:31.862 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:31.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:31.864 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:33.728 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:33.729 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:38.730 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:38.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:38.731 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:40.286 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:40.287 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:45.288 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:45.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:45.289 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:46.901 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:46.902 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:51.903 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:51.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:51.904 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:47:53.518 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:47:53.518 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:47:58.520 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:58.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:47:58.520 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:00.072 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:48:00.073 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:48:05.074 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:05.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:05.074 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:06.766 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:48:06.766 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:48:11.767 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:11.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:11.768 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:13.365 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:48:13.366 WARNING:rgw_multi.tests:bucket axxsjh-17 zone test-zone4 behind zone test-zone2: shard=0 master=00000000007.16.9 target= 2020-03-28T06:48:18.377 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... FAIL 2020-03-28T06:48:19.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:19.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:19.169 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:20.219 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:48:20.534 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585378099.132134_110.1","last_update":"2020-03-28 06:48:19.132134Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:48:20.534 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:48:20.534 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:20.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:20.535 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:21.809 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378099.132134_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:48:19.132134Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:48:21.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:48:21.809 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:48:21.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:21.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:21.810 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:23.436 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378099.132134_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:48:19.132134Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:48:23.437 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:48:23.437 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:48:23.437 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:23.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:23.438 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:25.101 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378099.132134_110.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:48:19.132134Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:48:25.102 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:48:25.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:25.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:25.711 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:26.325 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:30:58.296 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:48:26.331 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:30:58.303 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:48:26.568 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:30:58.540 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:48:27.330 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000003.3.10"}]} 2020-03-28T06:48:27.330 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-19 2020-03-28T06:48:27.330 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:27.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:27.331 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:28.726 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"gj3VVCKz6w0z2uRQERAX-QEHEyYhHwd","ns":""},"count":1},"inc_marker":{"position":"0#00000000003.3.10"}}}] 2020-03-28T06:48:28.726 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-19 2020-03-28T06:48:28.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:28.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:28.727 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:30.664 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000007.9.9"}]} 2020-03-28T06:48:30.664 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-19 2020-03-28T06:48:30.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:30.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:30.665 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:32.014 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"gj3VVCKz6w0z2uRQERAX-QEHEyYhHwd","ns":""},"count":1},"inc_marker":{"position":"00000000007.9.9"}}}] 2020-03-28T06:48:32.014 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-19 2020-03-28T06:48:32.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:32.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:32.015 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:33.316 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000007.9.9"}]} 2020-03-28T06:48:33.317 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-19 2020-03-28T06:48:33.317 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:33.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:33.318 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:35.140 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"gj3VVCKz6w0z2uRQERAX-QEHEyYhHwd","ns":""},"count":1},"inc_marker":{"position":"00000000007.9.9"}}}] 2020-03-28T06:48:35.141 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-19 2020-03-28T06:48:35.141 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:35.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:35.141 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:36.451 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000005.8.9"}]} 2020-03-28T06:48:36.452 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-19 2020-03-28T06:48:36.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:36.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:36.452 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:37.869 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"gj3VVCKz6w0z2uRQERAX-QEHEyYhHwd","ns":""},"count":1},"inc_marker":{"position":"00000000005.8.9"}}}] 2020-03-28T06:48:37.870 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-19 2020-03-28T06:48:37.870 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:37.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:37.871 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:39.344 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000005.8.9"}]} 2020-03-28T06:48:39.345 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-19 2020-03-28T06:48:39.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:39.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:39.346 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:40.726 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"gj3VVCKz6w0z2uRQERAX-QEHEyYhHwd","ns":""},"count":1},"inc_marker":{"position":"00000000005.8.9"}}}] 2020-03-28T06:48:40.726 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-19 2020-03-28T06:48:40.726 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:40.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:40.727 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:42.121 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000005.8.9"}]} 2020-03-28T06:48:42.122 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-19 2020-03-28T06:48:42.122 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:42.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:42.122 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:43.710 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:48:43.711 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:48:48.712 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:48.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:48.714 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:50.335 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:48:50.335 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:48:55.336 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:55.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:48:55.337 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:48:57.113 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:48:57.114 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:02.114 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:02.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:02.114 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:03.782 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:03.783 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:08.783 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:08.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:08.784 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:10.439 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:10.439 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:15.440 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:15.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:15.441 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:17.082 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:17.082 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:22.083 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:22.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:22.084 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:23.737 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:23.737 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:28.738 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:28.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:28.739 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:30.541 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:30.542 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:35.543 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:35.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:35.544 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:37.056 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:37.057 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:42.058 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:42.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:42.059 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:43.909 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:43.910 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:44.231 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:31:10.288 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:49:44.237 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:31:10.294 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:49:44.490 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:31:10.548 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T06:49:48.910 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:48.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:48.911 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:50.619 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:50.620 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:49:55.621 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:55.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:49:55.621 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:49:57.149 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:49:57.149 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:02.149 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:02.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:02.150 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:03.834 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:03.834 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:08.835 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:08.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:08.836 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:10.521 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:10.522 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:15.523 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:15.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:15.523 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:17.210 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:17.211 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:22.212 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:22.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:22.212 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:23.945 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:23.945 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:28.947 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:28.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:28.947 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:30.843 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:30.844 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:35.846 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:35.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:35.846 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:37.548 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:37.548 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:42.549 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:42.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:42.549 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:44.217 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:44.218 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:49.219 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:49.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:49.219 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:51.158 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:51.158 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:50:56.159 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:56.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:50:56.160 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:50:57.786 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:50:57.787 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:02.788 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:02.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:02.788 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:04.554 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:04.554 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:09.555 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:09.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:09.556 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:11.125 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:11.125 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:16.126 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:16.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:16.127 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:17.769 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:17.769 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:22.770 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:22.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:22.771 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:24.462 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:24.462 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:29.463 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:29.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:29.464 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:31.217 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:31.217 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:36.218 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:36.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:36.218 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:38.226 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:38.227 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:43.227 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:43.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:43.228 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:45.094 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:45.095 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:50.095 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:50.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:50.096 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:51.765 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:51.766 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:51:56.766 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:56.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:51:56.767 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:51:58.427 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:51:58.428 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:03.429 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:03.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:03.429 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:04.994 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:04.994 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:09.996 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:09.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:09.996 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:11.809 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:11.809 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:16.810 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:16.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:16.811 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:18.552 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:18.553 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:23.554 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:23.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:23.554 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:25.186 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:25.187 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:30.188 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:30.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:30.189 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:31.813 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:31.814 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:36.814 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:36.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:36.815 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:38.567 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:38.568 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:43.568 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:43.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:43.569 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:44.994 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:44.994 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:49.995 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:49.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:49.996 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:51.572 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:51.572 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:52:56.574 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:56.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:52:56.575 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:52:58.400 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:52:58.401 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:03.402 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:03.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:03.403 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:04.998 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:04.999 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:10.000 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:10.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:10.001 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:11.922 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:11.923 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:16.924 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:16.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:16.925 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:18.693 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:18.693 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:23.694 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:23.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:23.695 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:25.226 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:25.227 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:30.228 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:30.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:30.230 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:31.876 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:31.877 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:36.878 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:36.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:36.878 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:38.789 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:38.789 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:43.790 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:43.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:43.791 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:45.363 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:45.363 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:50.364 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:50.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:50.365 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:52.162 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:52.163 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:53:57.164 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:57.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:53:57.164 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:53:58.903 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:53:58.903 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:03.904 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:03.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:03.904 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:05.653 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:05.653 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:10.654 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:10.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:10.655 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:12.400 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:12.400 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:17.401 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:17.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:17.402 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:19.163 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:19.164 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:24.165 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:24.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:24.165 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:25.925 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:25.925 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:30.926 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:30.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:30.927 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:32.597 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:32.597 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:37.599 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:37.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:37.599 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:39.316 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:39.317 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:44.317 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:44.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:44.318 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:46.191 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:46.191 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:51.192 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:51.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:51.193 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:52.893 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:52.894 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:54:57.895 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:57.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:54:57.895 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:54:59.727 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:54:59.728 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:55:04.729 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:04.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:04.730 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:06.400 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:55:06.401 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:55:11.402 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:11.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:11.403 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:13.203 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:55:13.204 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:55:18.205 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:18.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:18.205 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:19.866 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:55:19.866 WARNING:rgw_multi.tests:bucket axxsjh-19 zone test-zone4 behind zone test-zone2: shard=0 master=00000000005.8.9 target= 2020-03-28T06:55:24.868 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... FAIL 2020-03-28T06:55:25.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:25.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:25.314 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:26.275 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T06:55:26.765 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585378525.227247_122.1","last_update":"2020-03-28 06:55:25.227247Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T06:55:26.766 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T06:55:26.766 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:26.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:26.766 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:28.146 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378525.227247_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:55:25.227247Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:55:28.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T06:55:28.147 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T06:55:28.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:28.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:28.148 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:29.492 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378525.227247_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:55:25.227247Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:55:29.492 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T06:55:29.492 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T06:55:29.493 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:29.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:29.493 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:31.230 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378525.227247_122.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 06:55:25.227247Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T06:55:31.230 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T06:55:31.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:31.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:31.640 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:32.822 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000002.4.5"}]} 2020-03-28T06:55:32.822 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-20 2020-03-28T06:55:32.823 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:32.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:32.823 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:34.318 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.4.5"}}}] 2020-03-28T06:55:34.318 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-20 2020-03-28T06:55:34.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:34.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:34.321 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:35.852 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000002.4.5"}]} 2020-03-28T06:55:35.852 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-20 2020-03-28T06:55:35.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:35.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:35.853 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:37.126 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.4.5"}}}] 2020-03-28T06:55:37.127 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-20 2020-03-28T06:55:37.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:37.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:37.127 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:38.418 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000002.4.5"}]} 2020-03-28T06:55:38.418 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-20 2020-03-28T06:55:38.418 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:38.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:38.419 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:40.324 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.4.5"}}}] 2020-03-28T06:55:40.324 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-20 2020-03-28T06:55:40.324 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:40.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:40.325 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:41.685 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.2.5"}]} 2020-03-28T06:55:41.685 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-20 2020-03-28T06:55:41.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:41.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:41.686 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:43.268 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.2.5"}}}] 2020-03-28T06:55:43.268 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-20 2020-03-28T06:55:43.268 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:43.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:43.269 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:44.646 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.2.5"}]} 2020-03-28T06:55:44.647 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-20 2020-03-28T06:55:44.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:44.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:44.647 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:46.107 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.2.5"}}}] 2020-03-28T06:55:46.107 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-20 2020-03-28T06:55:46.107 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:46.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:46.108 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:47.367 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.2.5"}]} 2020-03-28T06:55:47.367 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-20 2020-03-28T06:55:47.368 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:47.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:47.368 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:48.856 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:55:48.857 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:55:53.858 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:53.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:55:53.858 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:55:55.690 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:55:55.690 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:00.692 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:00.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:00.693 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:02.407 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:02.407 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:07.408 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:07.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:07.409 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:09.059 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:09.059 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:14.060 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:14.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:14.060 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:15.787 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:15.788 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:20.789 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:20.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:20.789 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:22.497 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:22.497 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:27.498 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:27.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:27.499 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:29.253 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:29.253 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:34.254 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:34.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:34.254 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:35.923 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:35.923 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:40.924 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:40.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:40.925 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:42.735 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:42.736 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:47.737 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:47.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:47.737 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:49.440 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:49.440 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:56:54.442 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:54.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:56:54.442 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:56:56.215 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:56:56.216 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:01.225 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:01.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:01.225 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:02.895 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:02.895 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:07.897 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:07.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:07.897 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:09.593 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:09.594 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:14.595 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:14.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:14.595 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:16.451 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:16.451 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:21.452 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:21.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:21.453 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:23.146 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:23.147 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:28.148 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:28.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:28.149 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:29.887 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:29.887 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:34.888 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:34.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:34.889 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:36.617 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:36.617 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:41.618 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:41.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:41.618 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:43.431 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:43.432 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:48.432 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:48.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:48.433 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:50.124 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:50.124 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:57:55.125 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:55.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:57:55.126 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:57:56.880 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:57:56.880 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:01.881 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:01.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:01.881 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:03.766 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:03.766 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:08.771 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:08.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:08.772 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:10.617 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:10.617 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:15.618 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:15.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:15.619 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:17.569 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:17.570 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:22.571 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:22.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:22.572 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:24.246 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:24.246 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:29.249 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:29.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:29.249 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:31.105 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:31.105 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:36.106 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:36.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:36.107 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:37.848 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:37.848 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:42.849 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:42.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:42.850 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:44.636 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:44.637 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:49.637 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:49.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:49.638 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:51.715 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:51.715 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:58:56.716 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:56.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:58:56.717 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:58:58.649 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:58:58.649 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:03.650 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:03.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:03.651 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:05.474 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:05.474 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:10.478 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:10.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:10.479 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:12.149 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:12.150 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:17.151 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:17.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:17.152 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:18.887 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:18.887 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:23.888 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:23.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:23.889 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:25.663 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:25.663 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:30.664 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:30.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:30.664 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:32.478 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:32.479 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:37.480 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:37.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:37.481 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:39.491 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:39.491 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:44.492 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:44.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:44.493 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:46.281 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:46.281 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:48.964 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:42:20.934 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:59:48.969 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:42:20.939 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:59:49.419 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:42:21.389 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T06:59:51.284 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:51.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:51.284 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:52.918 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:52.919 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T06:59:57.920 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:57.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T06:59:57.921 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T06:59:59.731 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T06:59:59.731 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:04.732 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:04.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:04.733 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:06.522 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:06.522 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:11.525 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:11.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:11.525 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:13.325 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:13.326 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:18.327 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:18.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:18.328 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:20.169 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:20.169 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:25.170 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:25.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:25.171 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:27.019 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:27.020 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:32.021 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:32.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:32.022 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:33.812 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:33.813 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:38.813 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:38.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:38.814 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:40.552 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:40.553 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:45.554 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:45.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:45.554 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:47.257 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:47.257 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:52.258 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:52.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:52.259 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:00:54.030 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:00:54.031 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:00:54.435 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:42:20.496 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:00:54.472 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:42:20.533 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:00:54.816 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:42:20.878 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:00:59.032 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:59.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:00:59.032 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:00.835 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:00.835 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:05.837 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:05.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:05.837 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:07.685 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:07.685 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:12.687 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:12.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:12.688 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:14.421 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:14.421 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:19.422 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:19.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:19.423 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:21.225 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:21.226 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:26.228 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:26.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:26.228 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:27.865 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:27.865 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:32.867 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:32.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:32.867 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:34.874 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:34.875 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:39.876 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:39.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:39.876 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:41.934 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:41.934 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:46.937 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:46.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:46.938 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:48.811 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:48.811 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:01:53.812 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:53.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:01:53.813 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:01:55.716 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:01:55.717 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:02:00.717 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:00.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:00.718 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:02.578 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:02:02.578 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:02:07.583 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:07.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:07.584 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:09.211 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:02:09.212 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:02:14.212 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:14.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:14.213 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:15.959 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:02:15.959 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:02:17.651 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:44:49.621 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:02:17.660 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:44:49.630 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:02:17.806 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:44:49.776 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:02:20.961 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:20.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:20.962 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:22.793 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:02:22.793 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:02:27.794 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:27.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:27.794 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:29.585 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:02:29.586 WARNING:rgw_multi.tests:bucket axxsjh-20 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.2.5 target= 2020-03-28T07:02:34.587 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2020-03-28T07:02:34.587 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-21 2020-03-28T07:02:35.211 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-22 2020-03-28T07:02:36.450 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-23 2020-03-28T07:02:38.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:38.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:38.638 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:39.683 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T07:02:39.937 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585378958.514875_140.1","last_update":"2020-03-28 07:02:38.514875Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:02:39.937 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:02:39.937 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:39.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:39.938 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:41.275 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378958.514875_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:02:38.514875Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:02:41.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:02:41.276 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:02:41.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:41.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:41.276 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:42.859 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378958.514875_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:02:38.514875Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:02:42.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:02:42.860 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:02:42.860 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:42.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:42.861 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:44.644 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585378958.514875_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:02:38.514875Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:02:44.644 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:02:49.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:49.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:49.561 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:52.929 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000004.9.11"}]} 2020-03-28T07:02:52.929 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-21 2020-03-28T07:02:52.929 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:52.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:52.930 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:54.499 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Wlbs2GP0lBcQmtODqBZX5ffnlBFtBzY","ns":""},"count":1},"inc_marker":{"position":"0#00000000004.9.11"}}}] 2020-03-28T07:02:54.500 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-21 2020-03-28T07:02:54.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:54.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:54.501 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:55.851 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000004.9.11"}]} 2020-03-28T07:02:55.851 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-21 2020-03-28T07:02:55.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:55.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:55.852 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:57.050 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"3HqZnWiURGFGZcaWmgdxHsAEN8n.N3M","ns":""},"count":1},"inc_marker":{"position":"00000000004.9.11"}}}] 2020-03-28T07:02:57.051 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-21 2020-03-28T07:02:57.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:57.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:57.051 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:58.192 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000004.9.11"}]} 2020-03-28T07:02:58.193 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-21 2020-03-28T07:02:58.193 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:58.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:58.194 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:02:59.993 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Wlbs2GP0lBcQmtODqBZX5ffnlBFtBzY","ns":""},"count":1},"inc_marker":{"position":"0#00000000004.9.11"}}}] 2020-03-28T07:02:59.994 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-21 2020-03-28T07:02:59.994 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:59.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:02:59.995 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:01.381 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000003.4.5"}]} 2020-03-28T07:03:01.381 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-21 2020-03-28T07:03:01.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:01.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:01.382 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:03.118 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Wlbs2GP0lBcQmtODqBZX5ffnlBFtBzY","ns":""},"count":1},"inc_marker":{"position":"0#00000000003.4.5"}}}] 2020-03-28T07:03:03.119 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-21 2020-03-28T07:03:03.119 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:03.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:03.119 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:04.609 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000003.4.5"}]} 2020-03-28T07:03:04.610 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-21 2020-03-28T07:03:04.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:04.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:04.610 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:06.416 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Wlbs2GP0lBcQmtODqBZX5ffnlBFtBzY","ns":""},"count":1},"inc_marker":{"position":"0#00000000003.4.5"}}}] 2020-03-28T07:03:06.416 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-21 2020-03-28T07:03:06.416 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:06.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:06.417 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:07.769 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000003.4.5"}]} 2020-03-28T07:03:07.769 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-21 2020-03-28T07:03:07.769 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:07.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:07.770 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:09.342 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:09.342 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:03:14.343 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:14.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:14.343 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:15.980 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:15.980 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:03:20.985 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:20.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:20.985 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:22.891 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:22.891 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:03:27.896 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:27.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:27.896 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:29.645 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:29.645 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:03:34.396 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:45:00.458 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:03:34.408 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:45:00.470 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:03:34.546 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:45:00.607 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:03:34.647 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:34.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:34.647 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:37.073 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:37.074 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:03:42.077 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:42.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:42.078 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:44.139 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:44.139 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:03:49.148 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:49.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:49.149 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:50.901 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:50.901 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:03:55.902 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:55.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:03:55.903 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:03:57.614 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:03:57.614 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:02.615 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:02.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:02.615 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:04.429 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:04.430 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:09.431 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:09.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:09.431 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:11.235 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:11.235 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:16.236 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:16.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:16.237 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:17.922 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:17.923 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:22.924 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:22.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:22.925 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:24.905 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:24.905 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:29.906 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:29.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:29.906 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:31.890 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:31.890 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:36.892 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:36.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:36.893 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:38.592 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:38.593 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:43.594 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:43.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:43.595 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:45.301 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:45.302 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:50.303 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:50.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:50.304 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:51.985 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:51.986 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:04:56.987 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:56.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:04:56.988 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:04:58.694 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:04:58.694 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:03.695 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:03.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:03.696 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:05.461 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:05.462 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:10.463 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:10.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:10.463 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:12.446 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:12.446 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:17.447 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:17.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:17.448 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:19.248 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:19.248 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:24.250 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:24.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:24.250 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:26.177 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:26.178 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:31.179 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:31.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:31.179 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:32.889 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:32.890 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:37.890 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:37.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:37.891 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:39.759 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:39.759 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:44.761 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:44.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:44.761 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:46.693 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:46.693 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:51.694 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:51.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:51.695 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:05:53.516 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:05:53.516 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:05:58.517 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:58.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:05:58.518 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:00.476 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:00.477 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:05.477 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:05.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:05.478 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:07.606 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:07.607 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:12.607 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:12.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:12.608 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:14.518 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:14.519 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:19.520 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:19.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:19.520 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:21.368 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:21.368 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:26.369 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:26.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:26.369 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:28.267 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:28.268 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:33.268 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:33.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:33.269 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:35.114 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:35.115 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:40.115 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:40.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:40.116 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:42.061 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:42.061 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:47.063 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:47.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:47.063 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:48.872 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:48.872 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:06:53.873 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:53.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:06:53.873 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:06:55.853 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:06:55.854 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:00.855 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:00.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:00.855 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:02.697 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:02.698 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:07.698 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:07.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:07.699 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:09.484 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:09.485 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:14.485 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:14.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:14.486 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:16.567 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:16.567 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:21.568 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:21.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:21.569 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:23.577 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:23.577 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:28.578 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:28.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:28.579 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:30.432 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:30.433 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:35.434 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:35.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:35.435 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:37.625 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:37.625 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:42.625 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:42.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:42.626 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:44.555 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:44.556 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:49.556 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:49.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:49.557 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:51.535 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:51.535 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:07:56.536 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:56.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:07:56.537 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:07:58.428 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:07:58.429 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:03.429 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:03.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:03.430 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:05.401 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:05.402 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:09.786 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:50:41.757 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:08:09.790 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:50:41.761 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:08:10.099 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:50:42.069 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:08:10.402 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:10.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:10.403 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:12.373 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:12.373 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:17.373 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:17.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:17.374 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:19.475 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:19.476 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:24.477 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:24.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:24.477 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:26.411 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:26.412 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:31.412 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:31.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:31.413 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:33.171 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:33.171 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:38.172 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:38.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:38.173 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:40.142 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:40.143 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:45.143 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:45.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:45.144 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:47.263 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:47.264 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:52.264 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:52.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:52.265 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:08:54.045 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:08:54.045 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:08:59.046 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:59.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:08:59.047 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:00.907 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:00.907 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:05.908 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:05.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:05.909 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:07.769 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:07.769 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:12.100 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:51:44.071 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:09:12.104 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:51:44.075 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:09:12.174 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:51:44.144 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:09:12.771 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:12.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:12.771 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:14.522 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:14.523 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:15.661 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:50:41.724 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:09:15.666 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:50:41.729 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:09:15.852 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:50:41.915 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:09:19.523 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:19.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:19.524 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:21.527 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:21.528 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:26.529 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:26.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:26.529 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:28.499 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:28.499 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:33.500 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:33.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:33.501 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:35.444 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:35.444 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:40.445 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:40.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:40.446 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:42.506 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:42.507 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:47.508 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:47.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:47.508 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:49.598 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:49.598 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:09:54.599 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:54.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:09:54.600 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:09:56.869 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:09:56.870 WARNING:rgw_multi.tests:bucket axxsjh-21 zone test-zone4 behind zone test-zone2: shard=0 master=00000000003.4.5 target= 2020-03-28T07:10:01.870 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... FAIL 2020-03-28T07:10:01.871 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-24 2020-03-28T07:10:02.340 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-25 2020-03-28T07:10:03.984 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-26 2020-03-28T07:10:07.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:07.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:07.756 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:08.840 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T07:10:09.116 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585379407.585682_187.1","last_update":"2020-03-28 07:10:07.585682Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:10:09.117 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:10:09.117 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:09.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:09.118 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:10.595 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379407.585682_187.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:10:07.585682Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:10:10.595 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:10:10.595 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:10:10.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:10.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:10.596 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:11.991 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379407.585682_187.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:10:07.585682Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:10:11.992 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:10:11.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:10:11.992 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:11.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:11.992 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:13.937 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379407.585682_187.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:10:07.585682Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:10:13.938 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:10:16.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:16.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:16.751 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:19.005 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000005.14.6"}]} 2020-03-28T07:10:19.006 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-24 2020-03-28T07:10:19.006 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:19.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:19.006 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:20.634 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"0#00000000005.14.6"}}}] 2020-03-28T07:10:20.635 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-24 2020-03-28T07:10:20.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:20.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:20.635 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:21.825 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000005.14.6"}]} 2020-03-28T07:10:21.825 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-24 2020-03-28T07:10:21.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:21.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:21.826 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"0#00000000005.14.6"}}}] 2020-03-28T07:10:23.319 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-24 2020-03-28T07:10:23.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:23.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:23.320 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:24.505 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000005.14.6"}]} 2020-03-28T07:10:24.506 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-24 2020-03-28T07:10:24.506 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:24.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:24.506 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:26.597 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"0#00000000005.14.6"}}}] 2020-03-28T07:10:26.598 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-24 2020-03-28T07:10:26.598 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:26.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:26.599 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:27.795 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000001.10.12"}]} 2020-03-28T07:10:27.796 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-24 2020-03-28T07:10:27.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:27.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:27.796 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:29.404 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"0#00000000001.10.12"}}}] 2020-03-28T07:10:29.405 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-24 2020-03-28T07:10:29.405 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:29.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:29.406 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:30.624 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000001.10.12"}]} 2020-03-28T07:10:30.625 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-24 2020-03-28T07:10:30.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:30.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:30.625 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:30.953 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:51:57.009 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:10:30.955 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:51:57.018 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:10:31.092 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:51:57.155 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:10:32.208 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"0#00000000001.10.12"}}}] 2020-03-28T07:10:32.208 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-24 2020-03-28T07:10:32.208 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:32.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:32.209 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:33.554 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000001.10.12"}]} 2020-03-28T07:10:33.554 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-24 2020-03-28T07:10:33.555 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:33.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:33.555 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:35.249 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:10:35.250 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:10:40.255 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:40.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:40.255 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:42.141 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:10:42.141 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:10:47.143 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:47.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:47.144 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:49.071 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:10:49.071 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:10:54.072 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:54.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:10:54.073 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:10:55.815 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:10:55.815 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:00.816 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:00.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:00.816 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:02.688 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:02.689 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:07.689 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:07.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:07.690 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:09.391 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:09.391 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:14.392 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:14.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:14.392 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:16.312 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:16.312 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:21.314 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:21.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:21.314 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:23.263 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:23.263 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:28.264 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:28.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:28.265 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:30.359 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:30.360 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:35.361 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:35.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:35.362 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:37.125 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:37.126 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:42.127 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:42.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:42.127 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:43.923 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:43.923 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:48.924 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:48.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:48.925 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:50.893 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:50.893 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:11:55.894 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:55.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:11:55.895 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:11:57.711 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:11:57.711 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:02.712 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:02.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:02.713 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:04.531 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:04.532 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:09.533 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:09.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:09.533 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:11.549 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:11.549 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:16.550 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:16.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:16.550 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:18.369 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:18.369 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:23.370 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:23.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:23.370 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:25.261 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:25.261 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:30.262 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:30.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:30.263 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:32.267 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:32.267 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:37.268 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:37.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:37.269 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:39.215 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:39.215 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:44.216 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:44.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:44.217 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:46.134 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:46.134 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:51.135 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:51.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:51.135 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:52.916 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:52.917 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:12:57.918 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:57.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:12:57.919 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:12:59.788 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:12:59.789 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:04.789 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:04.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:04.844 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:06.953 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:06.954 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:11.956 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:11.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:11.956 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:14.098 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:14.098 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:19.099 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:19.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:19.100 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:20.844 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:20.844 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:25.846 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:25.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:25.846 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:27.802 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:27.802 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:32.803 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:32.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:32.804 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:34.709 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:34.709 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:39.710 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:39.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:39.711 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:41.771 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:41.772 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:46.773 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:46.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:46.773 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:48.862 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:48.863 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:13:53.864 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:53.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:13:53.864 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:13:55.727 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:13:55.727 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:00.728 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:00.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:00.729 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:02.533 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:02.534 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:07.534 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:07.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:07.535 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:09.565 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:09.565 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:14.566 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:14.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:14.566 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:16.341 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:16.342 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:21.343 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:21.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:21.344 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:23.228 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:23.229 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:28.230 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:28.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:28.230 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:30.365 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:30.366 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:35.367 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:35.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:35.367 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:37.236 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:37.236 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:42.241 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:42.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:42.241 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:44.211 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:44.212 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:49.212 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:49.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:49.213 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:50.901 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:50.901 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:14:55.902 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:55.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:14:55.903 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:14:58.083 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:14:58.083 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:03.085 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:03.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:03.085 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:05.172 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:05.172 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:10.173 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:10.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:10.173 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:12.064 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:12.064 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:17.065 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:17.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:17.066 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:19.059 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:19.059 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:24.060 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:24.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:24.061 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:26.076 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:26.076 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:31.076 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:31.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:31.077 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:33.018 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:33.018 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:38.019 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:38.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:38.020 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:40.086 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:40.086 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:45.087 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:45.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:45.088 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:47.031 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:47.031 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:52.032 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:52.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:52.033 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:15:53.931 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:15:53.932 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:15:58.933 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:58.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:15:58.934 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:00.951 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:00.951 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:05.953 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:05.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:05.953 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:07.735 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:07.736 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:08.700 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:58:40.670 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:16:08.716 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:58:40.687 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:16:09.068 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:58:41.039 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:16:12.736 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:12.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:12.736 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:14.700 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:14.701 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:19.702 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:19.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:19.702 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:21.665 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:21.665 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:26.666 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:26.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:26.667 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:28.690 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:28.691 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:30.697 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:59:02.668 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:16:30.709 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:59:02.680 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:16:30.805 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:00:59:02.776 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:16:33.692 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:33.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:33.692 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:35.883 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:35.883 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:40.883 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:40.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:40.884 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:43.054 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:43.055 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:48.056 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:48.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:48.057 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:50.145 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:50.145 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:16:55.147 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:55.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:16:55.147 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:16:57.205 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:16:57.206 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:17:02.207 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:02.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:02.207 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:04.320 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:17:04.321 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:17:09.322 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:09.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:09.322 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:11.110 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:17:11.110 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:17:16.111 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:16.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:16.111 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:17.998 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:17:17.999 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:17:23.000 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:23.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:23.001 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:25.155 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:17:25.156 WARNING:rgw_multi.tests:bucket axxsjh-24 zone test-zone4 behind zone test-zone2: shard=0 master=00000000001.10.12 target= 2020-03-28T07:17:27.342 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:58:53.404 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:17:27.357 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:58:53.418 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:17:27.734 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:58:53.796 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:17:30.157 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... FAIL 2020-03-28T07:17:30.157 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-27 2020-03-28T07:17:30.763 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-28 2020-03-28T07:17:31.534 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-29 2020-03-28T07:17:34.405 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-03-28T07:17:34.405 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-30 2020-03-28T07:17:35.040 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-31 2020-03-28T07:17:36.257 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-32 2020-03-28T07:17:36.571 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:59:02.633 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:17:36.573 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:59:02.635 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:17:36.590 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:00:59:02.652 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:17:39.681 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-03-28T07:17:39.681 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-33 2020-03-28T07:17:40.252 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-34 2020-03-28T07:17:41.667 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-35 2020-03-28T07:17:46.257 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-03-28T07:17:46.257 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-36 2020-03-28T07:17:46.636 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-37 2020-03-28T07:17:48.015 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-38 2020-03-28T07:17:49.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:49.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:49.250 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:50.190 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T07:17:50.480 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585379868.672446_263.1","last_update":"2020-03-28 07:17:48.672446Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:17:50.481 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:17:50.482 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:50.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:50.482 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:51.803 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379868.672446_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:17:48.672446Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:17:51.803 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:17:51.804 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:17:51.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:51.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:51.804 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:53.279 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379868.672446_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:17:48.672446Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:17:53.279 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:17:53.279 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:17:53.279 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:53.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:17:53.280 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:17:55.249 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379868.672446_263.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:17:48.672446Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:17:55.250 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:17:57.139 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-03-28T07:17:57.139 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-39 2020-03-28T07:17:58.202 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-40 2020-03-28T07:17:59.977 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-41 2020-03-28T07:18:01.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:01.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:01.617 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:18:02.564 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=3894ea80-31ba-4896-8e09-e661b246b356 2020-03-28T07:18:02.853 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585379881.274012_275.1","last_update":"2020-03-28 07:18:01.274012Z"},{"marker":"1_1585376798.437614_173.1","last_update":"2020-03-28 06:26:38.437614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:18:02.853 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:18:02.854 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:02.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:02.854 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:18:04.309 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379881.274012_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:18:01.274012Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:18:04.310 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:18:04.310 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:18:04.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:04.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:04.311 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:18:05.573 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379881.274012_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:18:01.274012Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:18:05.573 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:18:05.573 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:18:05.574 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:05.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:18:05.574 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:18:07.525 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"3894ea80-31ba-4896-8e09-e661b246b356","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379881.274012_275.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:18:01.274012Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:18:07.525 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:18:07.526 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-03-28T07:18:07.526 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T07:18:07.555 INFO:tasks.rgw.c1.client.1.smithi156.stdout:2020-03-28 07:18:07.530 1b85f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 14748) UID: 0 2020-03-28T07:18:07.559 INFO:tasks.rgw.c1.client.1.smithi156.stdout:2020-03-28 07:18:07.534 4049040 -1 shutting down 2020-03-28T07:18:13.627 INFO:tasks.rgw.c1.client.1:Stopped 2020-03-28T07:18:13.627 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:18:13.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:18:13.628 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:18:14.085 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:18:14.083 7f8c21d44840 0 NOTICE: overriding master zone: 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T07:18:14.109 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:18:14.108 7f8c21d44840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T07:18:14.109 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:18:14.108 7f8c21d44840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T07:18:14.118 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:18:14.115 7f8c21d44840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T07:18:14.121 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:18:14.115 7f8c21d44840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T07:18:14.235 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:18:14.236 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:18:14.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:18:14.237 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:18:16.420 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"571a1648-7110-4ecb-afe2-66045679be41","epoch":1,"predecessor_uuid":"3894ea80-31ba-4896-8e09-e661b246b356","sync_status":["1_1585379881.274012_275.1","1_1585376798.437614_173.1","",""],"period_map":{"id":"571a1648-7110-4ecb-afe2-66045679be41","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":3} 2020-03-28T07:18:16.421 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-03-28T07:19:16.425 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-42 2020-03-28T07:19:17.435 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-43 2020-03-28T07:19:18.025 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:19:18.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:19:18.025 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:19:18.952 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=571a1648-7110-4ecb-afe2-66045679be41 2020-03-28T07:19:19.230 INFO:teuthology.orchestra.run.smithi168.stdout:[{"marker":"1_1585379957.915346_98.1","last_update":"2020-03-28 07:19:17.915346Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:19:19.230 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-03-28T07:19:19.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:19:19.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:19:19.231 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:19:20.613 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"571a1648-7110-4ecb-afe2-66045679be41","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585379957.915346_98.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:19:17.915346Z","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":25,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":41,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":14,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2020-03-28T07:19:20.614 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-03-28T07:19:20.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:19:20.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:19:20.614 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:19:21.111 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:19:21.109 7fa6f05f8840 0 NOTICE: overriding master zone: 2c0a3e73-a280-4ccf-960d-d2c86331b6ff 2020-03-28T07:19:21.120 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:19:21.118 7fa6f05f8840 0 failed reading obj info from .rgw.root:zone_info.2c0a3e73-a280-4ccf-960d-d2c86331b6ff: (2) No such file or directory 2020-03-28T07:19:21.120 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:19:21.118 7fa6f05f8840 0 WARNING: could not read zone params for zone id=2c0a3e73-a280-4ccf-960d-d2c86331b6ff name=test-zone2 2020-03-28T07:19:21.151 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:19:21.149 7fa6f05f8840 0 failed reading obj info from .rgw.root:zone_info.d8382bd8-febb-4ae8-8d2a-8fafdd38eff5: (2) No such file or directory 2020-03-28T07:19:21.151 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:19:21.149 7fa6f05f8840 0 WARNING: could not read zone params for zone id=d8382bd8-febb-4ae8-8d2a-8fafdd38eff5 name=test-zone4 2020-03-28T07:19:21.241 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:19:21.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:19:21.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:19:21.242 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:19:23.340 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":1,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:19:23.341 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-03-28T07:20:23.344 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-44 2020-03-28T07:20:23.955 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-45 2020-03-28T07:20:25.073 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-03-28T07:20:25.073 INFO:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-03-28T07:20:25.080 INFO:tasks.rgw.c1.client.1:Started 2020-03-28T07:20:25.080 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T07:20:25.182 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T07:20:25.182 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T07:20:25.186 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T07:20:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T07:20:26.189 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T07:20:26.272 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T07:20:26.272 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T07:20:26.277 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T07:20:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T07:20:28.277 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T07:20:28.357 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T07:20:28.357 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T07:20:28.361 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T07:20:28.361 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T07:20:32.362 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T07:20:32.431 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T07:20:32.431 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T07:20:32.434 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi156.front.sepia.ceph.com:8001; Connection refused 2020-03-28T07:20:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-03-28T07:20:40.435 INFO:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:8001 2020-03-28T07:20:40.488 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-03-28T07:20:40.488 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2020-03-28T07:20:40.764 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 780 0 --:--:-- --:--:-- --:--:-- 783 2020-03-28T07:20:40.764 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2020-03-28T07:20:40.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:40.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:40.765 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:20:41.954 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T07:20:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585380024.645876_62.1","last_update":"2020-03-28 07:20:24.645876Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:20:42.294 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:20:42.294 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:42.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:42.294 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:20:43.604 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380024.645876_62.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:20:24.645876Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:20:43.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:20:43.604 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:20:43.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:43.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:43.605 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:20:45.182 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380024.645876_62.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:20:24.645876Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:20:45.182 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:20:45.183 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:20:45.183 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:45.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:20:45.183 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:20:47.148 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380024.645876_62.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:20:24.645876Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:20:47.149 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:20:47.149 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-39 zones={test-zone2, test-zone1} 2020-03-28T07:20:47.427 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:47.546 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:47.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-39 zones={test-zone2, test-zone1} 2020-03-28T07:20:47.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-39 zones={test-zone3, test-zone1} 2020-03-28T07:20:48.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:48.568 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:48.723 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-39 zones={test-zone3, test-zone1} 2020-03-28T07:20:48.737 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-39 zones={test-zone1, test-zone2} 2020-03-28T07:20:48.867 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:48.941 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:49.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-39 zones={test-zone1, test-zone2} 2020-03-28T07:20:49.165 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-39 zones={test-zone3, test-zone2} 2020-03-28T07:20:49.299 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:49.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:49.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-39 zones={test-zone3, test-zone2} 2020-03-28T07:20:49.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-39 zones={test-zone1, test-zone3} 2020-03-28T07:20:49.735 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:49.907 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:50.066 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-39 zones={test-zone1, test-zone3} 2020-03-28T07:20:50.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-39 zones={test-zone2, test-zone3} 2020-03-28T07:20:50.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:50.298 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:50.607 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-39 zones={test-zone2, test-zone3} 2020-03-28T07:20:50.607 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-40 zones={test-zone2, test-zone1} 2020-03-28T07:20:51.019 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:51.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:51.402 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-40 zones={test-zone2, test-zone1} 2020-03-28T07:20:51.402 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-40 zones={test-zone3, test-zone1} 2020-03-28T07:20:51.604 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:51.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:51.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-40 zones={test-zone3, test-zone1} 2020-03-28T07:20:51.790 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-40 zones={test-zone1, test-zone2} 2020-03-28T07:20:51.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:52.025 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:52.316 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-40 zones={test-zone1, test-zone2} 2020-03-28T07:20:52.316 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-40 zones={test-zone3, test-zone2} 2020-03-28T07:20:52.417 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:52.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:52.628 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-40 zones={test-zone3, test-zone2} 2020-03-28T07:20:52.628 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-40 zones={test-zone1, test-zone3} 2020-03-28T07:20:52.774 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:52.820 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:52.961 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-40 zones={test-zone1, test-zone3} 2020-03-28T07:20:52.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-40 zones={test-zone2, test-zone3} 2020-03-28T07:20:53.186 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:53.257 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:53.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-40 zones={test-zone2, test-zone3} 2020-03-28T07:20:53.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-41 zones={test-zone2, test-zone1} 2020-03-28T07:20:53.848 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:53.967 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:54.246 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-41 zones={test-zone2, test-zone1} 2020-03-28T07:20:54.246 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-41 zones={test-zone3, test-zone1} 2020-03-28T07:20:54.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:54.540 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:54.695 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-41 zones={test-zone3, test-zone1} 2020-03-28T07:20:54.695 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-41 zones={test-zone1, test-zone2} 2020-03-28T07:20:54.843 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:54.909 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:55.097 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-41 zones={test-zone1, test-zone2} 2020-03-28T07:20:55.098 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-41 zones={test-zone3, test-zone2} 2020-03-28T07:20:55.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:55.601 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:55.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-41 zones={test-zone3, test-zone2} 2020-03-28T07:20:55.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-41 zones={test-zone1, test-zone3} 2020-03-28T07:20:55.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:55.957 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:56.104 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-41 zones={test-zone1, test-zone3} 2020-03-28T07:20:56.105 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-41 zones={test-zone2, test-zone3} 2020-03-28T07:20:56.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:56.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:56.753 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-41 zones={test-zone2, test-zone3} 2020-03-28T07:20:56.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-42 zones={test-zone2, test-zone1} 2020-03-28T07:20:56.882 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:56.936 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:57.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-42 zones={test-zone2, test-zone1} 2020-03-28T07:20:57.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-42 zones={test-zone3, test-zone1} 2020-03-28T07:20:57.278 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:57.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:57.548 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-42 zones={test-zone3, test-zone1} 2020-03-28T07:20:57.548 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-42 zones={test-zone1, test-zone2} 2020-03-28T07:20:57.684 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:57.748 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:57.919 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-42 zones={test-zone1, test-zone2} 2020-03-28T07:20:57.919 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-42 zones={test-zone3, test-zone2} 2020-03-28T07:20:58.082 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:58.173 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:58.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-42 zones={test-zone3, test-zone2} 2020-03-28T07:20:58.362 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-42 zones={test-zone1, test-zone3} 2020-03-28T07:20:58.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:58.561 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:58.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-42 zones={test-zone1, test-zone3} 2020-03-28T07:20:58.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-42 zones={test-zone2, test-zone3} 2020-03-28T07:20:58.870 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:58.929 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:59.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-42 zones={test-zone2, test-zone3} 2020-03-28T07:20:59.166 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-43 zones={test-zone2, test-zone1} 2020-03-28T07:20:59.322 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:59.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:59.635 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-43 zones={test-zone2, test-zone1} 2020-03-28T07:20:59.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-43 zones={test-zone3, test-zone1} 2020-03-28T07:20:59.778 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:20:59.830 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:20:59.995 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-43 zones={test-zone3, test-zone1} 2020-03-28T07:20:59.995 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-43 zones={test-zone1, test-zone2} 2020-03-28T07:21:00.127 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:00.198 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:00.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-43 zones={test-zone1, test-zone2} 2020-03-28T07:21:00.343 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-43 zones={test-zone3, test-zone2} 2020-03-28T07:21:00.499 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:00.565 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:00.750 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-43 zones={test-zone3, test-zone2} 2020-03-28T07:21:00.751 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-43 zones={test-zone1, test-zone3} 2020-03-28T07:21:00.881 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:00.935 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:01.073 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-43 zones={test-zone1, test-zone3} 2020-03-28T07:21:01.073 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-43 zones={test-zone2, test-zone3} 2020-03-28T07:21:01.237 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:01.315 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:01.489 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-43 zones={test-zone2, test-zone3} 2020-03-28T07:21:01.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-44 zones={test-zone2, test-zone1} 2020-03-28T07:21:01.622 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:01.664 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:01.872 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-44 zones={test-zone2, test-zone1} 2020-03-28T07:21:01.872 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-44 zones={test-zone3, test-zone1} 2020-03-28T07:21:01.968 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:02.007 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:02.136 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-44 zones={test-zone3, test-zone1} 2020-03-28T07:21:02.136 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-44 zones={test-zone1, test-zone2} 2020-03-28T07:21:02.262 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:02.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:02.536 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-44 zones={test-zone1, test-zone2} 2020-03-28T07:21:02.536 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-44 zones={test-zone3, test-zone2} 2020-03-28T07:21:02.662 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:02.739 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:02.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-44 zones={test-zone3, test-zone2} 2020-03-28T07:21:02.903 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-44 zones={test-zone1, test-zone3} 2020-03-28T07:21:03.001 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:03.060 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:03.203 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-44 zones={test-zone1, test-zone3} 2020-03-28T07:21:03.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-44 zones={test-zone2, test-zone3} 2020-03-28T07:21:03.381 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:03.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:03.639 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-44 zones={test-zone2, test-zone3} 2020-03-28T07:21:03.639 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-45 zones={test-zone2, test-zone1} 2020-03-28T07:21:03.765 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:03.817 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:04.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-45 zones={test-zone2, test-zone1} 2020-03-28T07:21:04.011 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-45 zones={test-zone3, test-zone1} 2020-03-28T07:21:04.141 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:04.197 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:04.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-45 zones={test-zone3, test-zone1} 2020-03-28T07:21:04.365 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-45 zones={test-zone1, test-zone2} 2020-03-28T07:21:04.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:04.574 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:04.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-45 zones={test-zone1, test-zone2} 2020-03-28T07:21:04.803 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-45 zones={test-zone3, test-zone2} 2020-03-28T07:21:04.963 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:05.049 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:05.256 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-45 zones={test-zone3, test-zone2} 2020-03-28T07:21:05.256 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-45 zones={test-zone1, test-zone3} 2020-03-28T07:21:05.422 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:05.471 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:05.628 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-45 zones={test-zone1, test-zone3} 2020-03-28T07:21:05.629 INFO:tasks.rgw_multi.zone_rados:comparing bucket=axxsjh-45 zones={test-zone2, test-zone3} 2020-03-28T07:21:05.770 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-03-28T07:21:05.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-03-28T07:21:05.988 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=axxsjh-45 zones={test-zone2, test-zone3} 2020-03-28T07:21:05.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:05.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:05.989 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:07.568 INFO:teuthology.orchestra.run.smithi156.stdout:[{"id":"1_1585376652.303723_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.303723698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585376652.440666_46.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.440666811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.148701_47.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.148701403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.258390_48.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.258390316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.295235_49.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.295235715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.366862_50.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.366862274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.742097_51.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.742097732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.843977_52.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.843977311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.889198_53.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.889198765Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.994586_54.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.994586485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.045207_55.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.045207216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.112783_56.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.112783859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.144565_57.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.144565273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.210517_58.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.210517506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.368401_67.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.368401761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.454520_68.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.454520379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.485914_69.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.485914075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.553433_70.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.553433913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.809723_71.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.809723515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.892857_72.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.892857302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.930302_73.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:05.930302745Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.016937_74.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:06.016937941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.616245_75.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.616245242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.769820_76.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.769820804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.872966_77.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:06.872966817Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"write"}}},{"id":"1_1585377247.211607_78.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:07.211607358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.480053_79.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.480053921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.569147_80.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.569147410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.604959_81.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.604959147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.673746_82.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.673746144Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.073816_83.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.073816872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.208756_84.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.208756363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.262168_85.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.262168424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.351984_86.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.351984044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.239692_87.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.239692278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.490927_88.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.490927292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.617122_89.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.617122529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.693263_90.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.693263278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377674.063856_91.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.063856502Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.255798_92.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.255798523Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.516416_93.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.516416634Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.656326_94.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.656326019Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.966446_95.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:14.966446599Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"write"}}},{"id":"1_1585377675.127242_96.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:15.127242200Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378098.552235_105.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.552235547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.645668_106.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.645668263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.717973_107.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.717973691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.825676_108.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.825676145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.930106_109.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.930106779Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"write"}}},{"id":"1_1585378099.132134_110.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:19.132134061Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378525.026438_119.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.026438340Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.119525_120.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.119525723Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378525.154750_121.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.154750972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.227247_122.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.227247267Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378954.886285_123.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:34.886285208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.005537_124.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:35.005537210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.044075_125.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.044075963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.113951_126.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.113951241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.717034_127.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.717034672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.837085_128.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.837085674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.876631_129.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.876631464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.974300_130.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.974300064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.806273_131.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.806273600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.874519_132.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.874519868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.904938_133.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.904938372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.962527_134.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.962527960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378957.415262_135.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.415262594Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.584073_136.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.584073810Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378957.715517_137.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.715517926Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.850225_138.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.850225621Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378958.196850_139.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.196850880Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"write"}}},{"id":"1_1585378958.514875_140.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.514875144Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379402.082674_164.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.082674021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.165724_165.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.165724850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.205132_166.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.205132358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.270963_167.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.270963399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.766427_168.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.766427962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.897891_169.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.897891401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379403.013350_170.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.013350905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"write"}}},{"id":"1_1585379403.263728_171.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.263728640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.150477_172.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.150477693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.217340_173.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.217340110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.252681_174.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.252681155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.441937_175.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.441937707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.841841_176.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.841841152Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"write"}}},{"id":"1_1585379404.939836_177.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.939836691Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379405.021631_178.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.021631740Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"write"}}},{"id":"1_1585379405.138798_179.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.138798684Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379405.372671_180.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.372671815Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"write"}}},{"id":"1_1585379405.906068_181.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.906068792Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379406.212039_182.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.212039589Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"write"}}},{"id":"1_1585379406.487900_183.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.487900992Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379406.872428_184.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:06.872428649Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"write"}}},{"id":"1_1585379407.158264_185.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.158264055Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379407.405416_186.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.405416952Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"write"}}},{"id":"1_1585379407.585682_187.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.585682617Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379850.371148_196.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.371148834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.460427_197.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.460427247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.545339_198.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.545339790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.668860_199.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.668860950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.979253_200.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:30.979253727Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.049602_201.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:31.049602101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.082167_202.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.082167494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.167145_203.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.167145985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.929991_204.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:31.929991834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.204743_205.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:32.204743108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.301008_206.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.301008778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.388854_207.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.388854734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.722625_208.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.722625250Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"write"}}},{"id":"1_1585379852.892466_209.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.892466771Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.190280_210.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.190280998Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"write"}}},{"id":"1_1585379853.451679_211.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.451679986Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.819439_212.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:33.819439713Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"write"}}},{"id":"1_1585379854.172629_213.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:34.172629545Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379854.583256_214.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.583256878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.664872_215.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.664872210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379854.740120_216.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.740120131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.889105_217.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.889105146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.647104_218.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.647104317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.731907_219.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.731907759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.782813_220.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.782813367Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.851677_221.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.851677140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.669353_222.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.669353521Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.742235_223.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.742235698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.777249_224.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.777249761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.840487_225.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.840487434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379857.227131_227.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.227131243Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"write"}}},{"id":"1_1585379857.545595_228.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.545595233Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379858.131541_229.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.131541166Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"write"}}},{"id":"1_1585379858.424567_230.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.424567718Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379859.203855_231.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.203855216Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"write"}}},{"id":"1_1585379859.381569_232.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.381569502Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379860.007665_234.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.007665223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.086952_235.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.086952164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.122074_236.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.122074070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.186751_237.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.186751289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.543598_238.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.543598918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.904647_239.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.904647375Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379861.031309_240.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.031309531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"write"}}},{"id":"1_1585379861.294598_241.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.294598857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.193860_242.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.193860827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.315249_243.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.315249242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.369104_244.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.369104369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.439339_245.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.439339099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.787211_246.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:42.787211587Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"write"}}},{"id":"1_1585379863.098274_247.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:43.098274295Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379864.309844_248.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.309844529Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"write"}}},{"id":"1_1585379864.567708_249.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.567708251Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379865.935373_250.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:45.935373074Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"write"}}},{"id":"1_1585379866.043999_251.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:46.043999793Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379866.362803_252.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.362803884Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.434580_253.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.434580664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379866.484419_254.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.484419326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.563791_255.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.563791882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.020349_256.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.020349101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.312591_257.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.312591356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.395334_258.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.395334459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.497469_259.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.497469060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.309560_260.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.309560719Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.426071_261.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.426071387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.506294_262.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.506294170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.672446_263.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.672446537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.418476_264.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.418476376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.557238_265.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.557238989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.646448_266.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.646448712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.943421_267.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.943421678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.459199_268.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.459199595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"write"}}},{"id":"1_1585379878.695408_269.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.695408966Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.956601_270.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:58.956601405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"write"}}},{"id":"1_1585379879.324506_271.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:59.324506383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379880.595995_272.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:00.595995959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.038833_273.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:01.038833931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379881.172789_274.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.172789120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.274012_275.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.274012862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376740.861003_77.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:40.861003069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"write"}}},{"id":"1_1585376742.706900_78.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:42.706900935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376743.543331_79.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:43.543331546Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"write"}}},{"id":"1_1585376744.020908_80.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:44.020908648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376749.252320_81.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:49.252320307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.065997_82.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:52.065997004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376752.396205_83.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.396205990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.464236_84.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.464236341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.210946_85.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.210946488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.347177_86.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.347177695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.402195_87.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.402195270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.492951_88.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.492951180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.315487_89.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.315487274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.391442_90.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.391442919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.457622_91.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.457622052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.551172_92.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.551172358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.182472_93.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.182472572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.357825_94.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.357825619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.511973_95.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.511973399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.667779_96.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.667779763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.340138_97.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.340138823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.454915_98.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.454915090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.494752_99.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.494752076Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.686952_100.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.686952911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.580122_101.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.580122454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.659843_102.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.659843259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.689935_103.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.689935725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.763938_104.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.763938293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376772.846391_105.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.846391689Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376772.953845_106.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.953845611Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.145320_107.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.145320320Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.206662_108.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.206662772Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.237606_109.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.237606068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.291875_110.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.291875555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376773.349896_111.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.349896163Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376773.492815_112.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.492815556Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.864643_113.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.864643930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.940990_114.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.940990102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.976369_115.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:13.976369166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.101523_116.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:14.101523605Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376774.228124_117.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.228124246Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376774.340781_118.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.340781699Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376774.665269_119.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.665269386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.731256_120.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.731256315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376774.812674_121.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.812674535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.896072_122.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.896072588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376775.027616_123.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.027616069Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376775.260905_124.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.260905974Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376775.927416_125.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:15.927416490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.150255_126.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.150255103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376776.204310_127.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.204310139Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.283073_128.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.283073483Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376776.345243_129.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.345243656Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376776.442968_130.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.442968228Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376777.171070_131.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.171070476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.251548_132.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.251548808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376777.313435_133.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.313435517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.473114_134.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.473114274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376777.546664_135.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.546664324Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376777.648190_136.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.648190953Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376778.457614_137.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.457614242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.530866_138.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.530866422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376778.596300_139.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.596300056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.723354_140.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.723354680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376778.889445_141.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.889445931Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376778.993727_142.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.993727222Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376779.554126_143.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.554126870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.615566_144.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.615566263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376779.648136_145.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.648136892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.700575_146.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.700575070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376779.755698_147.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.755698282Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376779.836329_148.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.836329369Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376780.412105_149.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.412105520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.497832_150.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.497832329Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376780.528435_151.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.528435294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.580072_152.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.580072140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376780.637374_153.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.637374040Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376780.715973_154.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.715973688Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376788.749165_156.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.749165145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"write"}}},{"id":"1_1585376788.820425_157.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.820425744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376788.893305_158.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:28.893305113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.034890_159.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:29.034890598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.439220_160.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.439220137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.703446_161.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.703446147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.751465_162.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.751465528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.844685_163.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.844685962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.493616_164.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.493616022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.560835_165.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.560835206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.595668_166.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.595668729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.668987_167.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.668987885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376797.170139_168.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.170139053Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.255629_169.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.255629310Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376797.661171_170.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.661171532Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.740373_171.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.740373426Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376798.256348_172.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.256348601Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376798.437614_173.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.437614170Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"complete"}}}] 2020-03-28T07:21:07.572 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:07.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:07.573 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:09.378 INFO:teuthology.orchestra.run.smithi168.stdout:[{"id":"1_1585376652.303723_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.303723698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585376652.440666_46.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.440666811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.148701_47.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.148701403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.258390_48.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.258390316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.295235_49.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.295235715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.366862_50.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.366862274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.742097_51.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.742097732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.843977_52.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.843977311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.889198_53.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.889198765Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.994586_54.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.994586485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.045207_55.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.045207216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.112783_56.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.112783859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.144565_57.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.144565273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.210517_58.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.210517506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.368401_67.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.368401761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.454520_68.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.454520379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.485914_69.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.485914075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.553433_70.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.553433913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.809723_71.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.809723515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.892857_72.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.892857302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.930302_73.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:05.930302745Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.016937_74.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:06.016937941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.616245_75.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.616245242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.769820_76.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.769820804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.872966_77.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:06.872966817Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"write"}}},{"id":"1_1585377247.211607_78.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:07.211607358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.480053_79.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.480053921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.569147_80.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.569147410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.604959_81.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.604959147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.673746_82.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.673746144Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.073816_83.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.073816872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.208756_84.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.208756363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.262168_85.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.262168424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.351984_86.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.351984044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.239692_87.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.239692278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.490927_88.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.490927292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.617122_89.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.617122529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.693263_90.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.693263278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377674.063856_91.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.063856502Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.255798_92.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.255798523Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.516416_93.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.516416634Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.656326_94.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.656326019Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.966446_95.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:14.966446599Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"write"}}},{"id":"1_1585377675.127242_96.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:15.127242200Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378098.552235_105.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.552235547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.645668_106.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.645668263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.717973_107.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.717973691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.825676_108.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.825676145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.930106_109.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.930106779Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"write"}}},{"id":"1_1585378099.132134_110.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:19.132134061Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378525.026438_119.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.026438340Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.119525_120.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.119525723Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378525.154750_121.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.154750972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.227247_122.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.227247267Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378954.886285_123.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:34.886285208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.005537_124.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:35.005537210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.044075_125.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.044075963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.113951_126.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.113951241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.717034_127.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.717034672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.837085_128.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.837085674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.876631_129.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.876631464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.974300_130.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.974300064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.806273_131.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.806273600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.874519_132.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.874519868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.904938_133.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.904938372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.962527_134.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.962527960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378957.415262_135.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.415262594Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.584073_136.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.584073810Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378957.715517_137.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.715517926Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.850225_138.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.850225621Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378958.196850_139.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.196850880Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"write"}}},{"id":"1_1585378958.514875_140.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.514875144Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379402.082674_164.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.082674021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.165724_165.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.165724850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.205132_166.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.205132358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.270963_167.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.270963399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.766427_168.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.766427962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.897891_169.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.897891401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379403.013350_170.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.013350905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"write"}}},{"id":"1_1585379403.263728_171.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.263728640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.150477_172.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.150477693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.217340_173.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.217340110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.252681_174.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.252681155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.441937_175.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.441937707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.841841_176.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.841841152Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"write"}}},{"id":"1_1585379404.939836_177.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.939836691Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379405.021631_178.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.021631740Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"write"}}},{"id":"1_1585379405.138798_179.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.138798684Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379405.372671_180.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.372671815Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"write"}}},{"id":"1_1585379405.906068_181.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.906068792Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379406.212039_182.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.212039589Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"write"}}},{"id":"1_1585379406.487900_183.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.487900992Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379406.872428_184.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:06.872428649Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"write"}}},{"id":"1_1585379407.158264_185.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.158264055Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379407.405416_186.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.405416952Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"write"}}},{"id":"1_1585379407.585682_187.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.585682617Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379850.371148_196.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.371148834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.460427_197.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.460427247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.545339_198.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.545339790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.668860_199.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.668860950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.979253_200.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:30.979253727Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.049602_201.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:31.049602101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.082167_202.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.082167494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.167145_203.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.167145985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.929991_204.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:31.929991834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.204743_205.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:32.204743108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.301008_206.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.301008778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.388854_207.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.388854734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.722625_208.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.722625250Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"write"}}},{"id":"1_1585379852.892466_209.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.892466771Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.190280_210.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.190280998Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"write"}}},{"id":"1_1585379853.451679_211.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.451679986Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.819439_212.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:33.819439713Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"write"}}},{"id":"1_1585379854.172629_213.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:34.172629545Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379854.583256_214.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.583256878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.664872_215.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.664872210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379854.740120_216.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.740120131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.889105_217.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.889105146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.647104_218.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.647104317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.731907_219.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.731907759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.782813_220.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.782813367Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.851677_221.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.851677140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.669353_222.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.669353521Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.742235_223.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.742235698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.777249_224.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.777249761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.840487_225.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.840487434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379857.227131_227.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.227131243Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"write"}}},{"id":"1_1585379857.545595_228.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.545595233Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379858.131541_229.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.131541166Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"write"}}},{"id":"1_1585379858.424567_230.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.424567718Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379859.203855_231.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.203855216Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"write"}}},{"id":"1_1585379859.381569_232.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.381569502Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379860.007665_234.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.007665223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.086952_235.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.086952164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.122074_236.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.122074070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.186751_237.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.186751289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.543598_238.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.543598918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.904647_239.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.904647375Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379861.031309_240.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.031309531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"write"}}},{"id":"1_1585379861.294598_241.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.294598857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.193860_242.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.193860827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.315249_243.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.315249242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.369104_244.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.369104369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.439339_245.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.439339099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.787211_246.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:42.787211587Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"write"}}},{"id":"1_1585379863.098274_247.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:43.098274295Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379864.309844_248.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.309844529Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"write"}}},{"id":"1_1585379864.567708_249.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.567708251Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379865.935373_250.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:45.935373074Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"write"}}},{"id":"1_1585379866.043999_251.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:46.043999793Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379866.362803_252.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.362803884Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.434580_253.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.434580664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379866.484419_254.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.484419326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.563791_255.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.563791882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.020349_256.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.020349101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.312591_257.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.312591356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.395334_258.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.395334459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.497469_259.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.497469060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.309560_260.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.309560719Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.426071_261.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.426071387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.506294_262.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.506294170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.672446_263.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.672446537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.418476_264.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.418476376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.557238_265.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.557238989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.646448_266.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.646448712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.943421_267.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.943421678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.459199_268.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.459199595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"write"}}},{"id":"1_1585379878.695408_269.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.695408966Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.956601_270.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:58.956601405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"write"}}},{"id":"1_1585379879.324506_271.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:59.324506383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379880.595995_272.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:00.595995959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.038833_273.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:01.038833931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379881.172789_274.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.172789120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.274012_275.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.274012862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376740.861003_77.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:40.861003069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"write"}}},{"id":"1_1585376742.706900_78.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:42.706900935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376743.543331_79.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:43.543331546Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"write"}}},{"id":"1_1585376744.020908_80.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:44.020908648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376749.252320_81.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:49.252320307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.065997_82.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:52.065997004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376752.396205_83.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.396205990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.464236_84.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.464236341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.210946_85.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.210946488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.347177_86.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.347177695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.402195_87.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.402195270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.492951_88.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.492951180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.315487_89.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.315487274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.391442_90.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.391442919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.457622_91.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.457622052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.551172_92.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.551172358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.182472_93.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.182472572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.357825_94.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.357825619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.511973_95.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.511973399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.667779_96.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.667779763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.340138_97.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.340138823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.454915_98.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.454915090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.494752_99.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.494752076Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.686952_100.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.686952911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.580122_101.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.580122454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.659843_102.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.659843259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.689935_103.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.689935725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.763938_104.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.763938293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376772.846391_105.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.846391689Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376772.953845_106.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.953845611Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.145320_107.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.145320320Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.206662_108.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.206662772Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.237606_109.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.237606068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.291875_110.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.291875555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376773.349896_111.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.349896163Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376773.492815_112.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.492815556Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.864643_113.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.864643930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.940990_114.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.940990102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.976369_115.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:13.976369166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.101523_116.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:14.101523605Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376774.228124_117.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.228124246Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376774.340781_118.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.340781699Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376774.665269_119.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.665269386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.731256_120.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.731256315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376774.812674_121.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.812674535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.896072_122.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.896072588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376775.027616_123.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.027616069Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376775.260905_124.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.260905974Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376775.927416_125.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:15.927416490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.150255_126.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.150255103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376776.204310_127.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.204310139Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.283073_128.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.283073483Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376776.345243_129.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.345243656Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376776.442968_130.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.442968228Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376777.171070_131.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.171070476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.251548_132.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.251548808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376777.313435_133.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.313435517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.473114_134.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.473114274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376777.546664_135.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.546664324Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376777.648190_136.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.648190953Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376778.457614_137.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.457614242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.530866_138.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.530866422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376778.596300_139.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.596300056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.723354_140.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.723354680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376778.889445_141.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.889445931Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376778.993727_142.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.993727222Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376779.554126_143.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.554126870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.615566_144.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.615566263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376779.648136_145.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.648136892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.700575_146.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.700575070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376779.755698_147.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.755698282Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376779.836329_148.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.836329369Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376780.412105_149.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.412105520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.497832_150.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.497832329Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376780.528435_151.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.528435294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.580072_152.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.580072140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376780.637374_153.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.637374040Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376780.715973_154.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.715973688Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376788.749165_156.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.749165145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"write"}}},{"id":"1_1585376788.820425_157.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.820425744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376788.893305_158.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:28.893305113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.034890_159.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:29.034890598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.439220_160.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.439220137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.703446_161.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.703446147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.751465_162.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.751465528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.844685_163.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.844685962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.493616_164.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.493616022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.560835_165.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.560835206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.595668_166.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.595668729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.668987_167.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.668987885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376797.170139_168.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.170139053Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.255629_169.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.255629310Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376797.661171_170.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.661171532Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.740373_171.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.740373426Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376798.256348_172.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.256348601Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376798.437614_173.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.437614170Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"complete"}}}] 2020-03-28T07:21:09.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:09.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:09.388 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:11.065 INFO:teuthology.orchestra.run.smithi156.stdout:[{"id":"1_1585376652.303723_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.303723698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585376652.440666_46.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.440666811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.148701_47.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.148701403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.258390_48.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.258390316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.295235_49.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.295235715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.366862_50.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.366862274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.742097_51.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.742097732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.843977_52.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.843977311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.889198_53.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.889198765Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.994586_54.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.994586485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.045207_55.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.045207216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.112783_56.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.112783859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.144565_57.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.144565273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.210517_58.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.210517506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.368401_67.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.368401761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.454520_68.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.454520379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.485914_69.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.485914075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.553433_70.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.553433913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.809723_71.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.809723515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.892857_72.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.892857302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.930302_73.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:05.930302745Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.016937_74.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:06.016937941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.616245_75.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.616245242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.769820_76.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.769820804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.872966_77.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:06.872966817Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"write"}}},{"id":"1_1585377247.211607_78.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:07.211607358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.480053_79.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.480053921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.569147_80.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.569147410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.604959_81.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.604959147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.673746_82.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.673746144Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.073816_83.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.073816872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.208756_84.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.208756363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.262168_85.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.262168424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.351984_86.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.351984044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.239692_87.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.239692278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.490927_88.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.490927292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.617122_89.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.617122529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.693263_90.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.693263278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377674.063856_91.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.063856502Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.255798_92.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.255798523Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.516416_93.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.516416634Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.656326_94.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.656326019Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.966446_95.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:14.966446599Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"write"}}},{"id":"1_1585377675.127242_96.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:15.127242200Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378098.552235_105.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.552235547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.645668_106.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.645668263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.717973_107.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.717973691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.825676_108.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.825676145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.930106_109.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.930106779Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"write"}}},{"id":"1_1585378099.132134_110.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:19.132134061Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378525.026438_119.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.026438340Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.119525_120.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.119525723Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378525.154750_121.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.154750972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.227247_122.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.227247267Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378954.886285_123.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:34.886285208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.005537_124.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:35.005537210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.044075_125.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.044075963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.113951_126.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.113951241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.717034_127.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.717034672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.837085_128.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.837085674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.876631_129.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.876631464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.974300_130.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.974300064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.806273_131.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.806273600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.874519_132.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.874519868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.904938_133.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.904938372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.962527_134.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.962527960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378957.415262_135.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.415262594Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.584073_136.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.584073810Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378957.715517_137.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.715517926Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.850225_138.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.850225621Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378958.196850_139.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.196850880Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"write"}}},{"id":"1_1585378958.514875_140.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.514875144Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379402.082674_164.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.082674021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.165724_165.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.165724850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.205132_166.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.205132358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.270963_167.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.270963399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.766427_168.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.766427962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.897891_169.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.897891401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379403.013350_170.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.013350905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"write"}}},{"id":"1_1585379403.263728_171.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.263728640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.150477_172.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.150477693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.217340_173.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.217340110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.252681_174.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.252681155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.441937_175.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.441937707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.841841_176.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.841841152Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"write"}}},{"id":"1_1585379404.939836_177.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.939836691Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379405.021631_178.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.021631740Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"write"}}},{"id":"1_1585379405.138798_179.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.138798684Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379405.372671_180.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.372671815Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"write"}}},{"id":"1_1585379405.906068_181.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.906068792Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379406.212039_182.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.212039589Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"write"}}},{"id":"1_1585379406.487900_183.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.487900992Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379406.872428_184.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:06.872428649Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"write"}}},{"id":"1_1585379407.158264_185.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.158264055Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379407.405416_186.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.405416952Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"write"}}},{"id":"1_1585379407.585682_187.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.585682617Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379850.371148_196.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.371148834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.460427_197.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.460427247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.545339_198.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.545339790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.668860_199.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.668860950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.979253_200.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:30.979253727Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.049602_201.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:31.049602101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.082167_202.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.082167494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.167145_203.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.167145985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.929991_204.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:31.929991834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.204743_205.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:32.204743108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.301008_206.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.301008778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.388854_207.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.388854734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.722625_208.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.722625250Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"write"}}},{"id":"1_1585379852.892466_209.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.892466771Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.190280_210.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.190280998Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"write"}}},{"id":"1_1585379853.451679_211.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.451679986Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.819439_212.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:33.819439713Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"write"}}},{"id":"1_1585379854.172629_213.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:34.172629545Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379854.583256_214.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.583256878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.664872_215.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.664872210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379854.740120_216.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.740120131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.889105_217.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.889105146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.647104_218.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.647104317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.731907_219.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.731907759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.782813_220.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.782813367Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.851677_221.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.851677140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.669353_222.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.669353521Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.742235_223.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.742235698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.777249_224.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.777249761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.840487_225.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.840487434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379857.227131_227.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.227131243Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"write"}}},{"id":"1_1585379857.545595_228.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.545595233Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379858.131541_229.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.131541166Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"write"}}},{"id":"1_1585379858.424567_230.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.424567718Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379859.203855_231.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.203855216Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"write"}}},{"id":"1_1585379859.381569_232.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.381569502Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379860.007665_234.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.007665223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.086952_235.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.086952164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.122074_236.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.122074070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.186751_237.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.186751289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.543598_238.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.543598918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.904647_239.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.904647375Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379861.031309_240.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.031309531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"write"}}},{"id":"1_1585379861.294598_241.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.294598857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.193860_242.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.193860827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.315249_243.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.315249242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.369104_244.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.369104369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.439339_245.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.439339099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.787211_246.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:42.787211587Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"write"}}},{"id":"1_1585379863.098274_247.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:43.098274295Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379864.309844_248.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.309844529Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"write"}}},{"id":"1_1585379864.567708_249.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.567708251Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379865.935373_250.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:45.935373074Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"write"}}},{"id":"1_1585379866.043999_251.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:46.043999793Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379866.362803_252.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.362803884Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.434580_253.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.434580664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379866.484419_254.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.484419326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.563791_255.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.563791882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.020349_256.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.020349101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.312591_257.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.312591356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.395334_258.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.395334459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.497469_259.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.497469060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.309560_260.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.309560719Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.426071_261.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.426071387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.506294_262.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.506294170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.672446_263.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.672446537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.418476_264.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.418476376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.557238_265.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.557238989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.646448_266.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.646448712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.943421_267.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.943421678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.459199_268.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.459199595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"write"}}},{"id":"1_1585379878.695408_269.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.695408966Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.956601_270.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:58.956601405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"write"}}},{"id":"1_1585379879.324506_271.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:59.324506383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379880.595995_272.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:00.595995959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.038833_273.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:01.038833931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379881.172789_274.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.172789120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.274012_275.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.274012862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376740.861003_77.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:40.861003069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"write"}}},{"id":"1_1585376742.706900_78.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:42.706900935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376743.543331_79.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:43.543331546Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"write"}}},{"id":"1_1585376744.020908_80.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:44.020908648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376749.252320_81.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:49.252320307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.065997_82.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:52.065997004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376752.396205_83.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.396205990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.464236_84.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.464236341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.210946_85.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.210946488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.347177_86.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.347177695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.402195_87.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.402195270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.492951_88.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.492951180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.315487_89.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.315487274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.391442_90.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.391442919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.457622_91.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.457622052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.551172_92.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.551172358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.182472_93.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.182472572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.357825_94.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.357825619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.511973_95.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.511973399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.667779_96.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.667779763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.340138_97.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.340138823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.454915_98.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.454915090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.494752_99.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.494752076Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.686952_100.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.686952911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.580122_101.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.580122454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.659843_102.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.659843259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.689935_103.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.689935725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.763938_104.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.763938293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376772.846391_105.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.846391689Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376772.953845_106.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.953845611Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.145320_107.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.145320320Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.206662_108.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.206662772Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.237606_109.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.237606068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.291875_110.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.291875555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376773.349896_111.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.349896163Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376773.492815_112.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.492815556Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.864643_113.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.864643930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.940990_114.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.940990102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.976369_115.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:13.976369166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.101523_116.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:14.101523605Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376774.228124_117.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.228124246Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376774.340781_118.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.340781699Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376774.665269_119.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.665269386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.731256_120.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.731256315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376774.812674_121.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.812674535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.896072_122.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.896072588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376775.027616_123.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.027616069Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376775.260905_124.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.260905974Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376775.927416_125.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:15.927416490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.150255_126.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.150255103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376776.204310_127.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.204310139Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.283073_128.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.283073483Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376776.345243_129.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.345243656Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376776.442968_130.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.442968228Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376777.171070_131.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.171070476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.251548_132.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.251548808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376777.313435_133.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.313435517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.473114_134.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.473114274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376777.546664_135.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.546664324Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376777.648190_136.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.648190953Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376778.457614_137.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.457614242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.530866_138.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.530866422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376778.596300_139.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.596300056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.723354_140.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.723354680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376778.889445_141.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.889445931Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376778.993727_142.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.993727222Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376779.554126_143.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.554126870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.615566_144.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.615566263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376779.648136_145.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.648136892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.700575_146.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.700575070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376779.755698_147.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.755698282Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376779.836329_148.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.836329369Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376780.412105_149.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.412105520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.497832_150.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.497832329Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376780.528435_151.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.528435294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.580072_152.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.580072140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376780.637374_153.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.637374040Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376780.715973_154.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.715973688Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376788.749165_156.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.749165145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"write"}}},{"id":"1_1585376788.820425_157.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.820425744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376788.893305_158.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:28.893305113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.034890_159.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:29.034890598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.439220_160.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.439220137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.703446_161.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.703446147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.751465_162.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.751465528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.844685_163.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.844685962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.493616_164.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.493616022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.560835_165.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.560835206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.595668_166.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.595668729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.668987_167.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.668987885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376797.170139_168.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.170139053Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.255629_169.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.255629310Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376797.661171_170.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.661171532Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.740373_171.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.740373426Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376798.256348_172.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.256348601Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376798.437614_173.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.437614170Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"complete"}}}] 2020-03-28T07:21:11.070 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:11.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:11.071 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:12.989 INFO:teuthology.orchestra.run.smithi168.stdout:[{"id":"1_1585376652.303723_45.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.303723698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585376652.440666_46.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-03-28 06:24:12.440666811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rbu2xyXEw-VFuJuN8kC-zWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.148701_47.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.148701403Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.258390_48.1","section":"bucket.instance","name":"axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","timestamp":"2020-03-28 06:26:45.258390316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7CwAkYLkE3o1UPKbbkRzB1u","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.295235_49.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.295235715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.366862_50.1","section":"bucket","name":"axxsjh-10","timestamp":"2020-03-28 06:26:45.366862274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8LnE1YWd0Zy2ehy-2IiT16d","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.742097_51.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.742097732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.843977_52.1","section":"bucket.instance","name":"axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","timestamp":"2020-03-28 06:26:45.843977311Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qiYcca6jM2ZBqwBwARALvd7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376805.889198_53.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.889198765Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"write"}}},{"id":"1_1585376805.994586_54.1","section":"bucket","name":"axxsjh-11","timestamp":"2020-03-28 06:26:45.994586485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xv4x_L80nuhDcwax0qLitrs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.045207_55.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.045207216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.112783_56.1","section":"bucket.instance","name":"axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","timestamp":"2020-03-28 06:26:47.112783859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UFwUVGgoVCbq2KsC8DhNVeO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376807.144565_57.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.144565273Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"write"}}},{"id":"1_1585376807.210517_58.1","section":"bucket","name":"axxsjh-12","timestamp":"2020-03-28 06:26:47.210517506Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cy8oi5LFggBZix-2zIaUy6L","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.368401_67.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.368401761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.454520_68.1","section":"bucket.instance","name":"axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","timestamp":"2020-03-28 06:34:05.454520379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eyk3hXrNVZz80aDJzYx92Jf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.485914_69.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.485914075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.553433_70.1","section":"bucket","name":"axxsjh-13","timestamp":"2020-03-28 06:34:05.553433913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LWR-8enLIRGq7eCUIBcRNZM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.809723_71.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.809723515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"write"}}},{"id":"1_1585377245.892857_72.1","section":"bucket.instance","name":"axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","timestamp":"2020-03-28 06:34:05.892857302Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QtCtFqPdt60cGpSCJaaJcU1","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377245.930302_73.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:05.930302745Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.016937_74.1","section":"bucket","name":"axxsjh-14","timestamp":"2020-03-28 06:34:06.016937941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhimbKWt-C26P_0aGiJAT_9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.616245_75.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.616245242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"write"}}},{"id":"1_1585377246.769820_76.1","section":"bucket.instance","name":"axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","timestamp":"2020-03-28 06:34:06.769820804Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6PFa4-0adnd1dLtSAIv-4-7","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377246.872966_77.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:06.872966817Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"write"}}},{"id":"1_1585377247.211607_78.1","section":"bucket","name":"axxsjh-15","timestamp":"2020-03-28 06:34:07.211607358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlYebmliS2K7bq7a86Tr0ny","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.480053_79.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.480053921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.569147_80.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:11.569147410Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377671.604959_81.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.604959147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"write"}}},{"id":"1_1585377671.673746_82.1","section":"bucket","name":"axxsjh-16","timestamp":"2020-03-28 06:41:11.673746144Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eS-4b1lQjVctuL8rEgubwI9","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.073816_83.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.073816872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.208756_84.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:12.208756363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377672.262168_85.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.262168424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"write"}}},{"id":"1_1585377672.351984_86.1","section":"bucket","name":"axxsjh-17","timestamp":"2020-03-28 06:41:12.351984044Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_50SXddrvRsUNG66zpLeqs4-","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.239692_87.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.239692278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.490927_88.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:13.490927292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377673.617122_89.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.617122529Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"write"}}},{"id":"1_1585377673.693263_90.1","section":"bucket","name":"axxsjh-18","timestamp":"2020-03-28 06:41:13.693263278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U9XaacRA15n9atEpDPlzQ_p","ver":1},"status":{"status":"complete"}}},{"id":"1_1585377674.063856_91.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.063856502Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.255798_92.1","section":"bucket.instance","name":"axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","timestamp":"2020-03-28 06:41:14.255798523Z","data":{"read_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":1},"write_version":{"tag":"_53G7ZDI8CJbvZUSp-lnHy6k","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.516416_93.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.516416634Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"write"}}},{"id":"1_1585377674.656326_94.1","section":"bucket.instance","name":"axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","timestamp":"2020-03-28 06:41:14.656326019Z","data":{"read_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":1},"write_version":{"tag":"_OPfOTeEoWoq25sxhVtlCjGc","ver":2},"status":{"status":"complete"}}},{"id":"1_1585377674.966446_95.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:14.966446599Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"write"}}},{"id":"1_1585377675.127242_96.1","section":"bucket.instance","name":"axxsjh-18:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27","timestamp":"2020-03-28 06:41:15.127242200Z","data":{"read_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":1},"write_version":{"tag":"_D-BgPvdhYbeQGpTiDohSRUz","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378098.552235_105.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.552235547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.645668_106.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.645668263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.717973_107.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.717973691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"write"}}},{"id":"1_1585378098.825676_108.1","section":"bucket","name":"axxsjh-19","timestamp":"2020-03-28 06:48:18.825676145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XVGQcBMJXlEIFZ8ASVNvfRO","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378098.930106_109.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:18.930106779Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"write"}}},{"id":"1_1585378099.132134_110.1","section":"bucket.instance","name":"axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","timestamp":"2020-03-28 06:48:19.132134061Z","data":{"read_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":1},"write_version":{"tag":"_QjuedZpqwCNOIzhcHoi8CP3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378525.026438_119.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.026438340Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.119525_120.1","section":"bucket.instance","name":"axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","timestamp":"2020-03-28 06:55:25.119525723Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dtVfgTu_Bl0ZRxBT4dyDHBs","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378525.154750_121.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.154750972Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"write"}}},{"id":"1_1585378525.227247_122.1","section":"bucket","name":"axxsjh-20","timestamp":"2020-03-28 06:55:25.227247267Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_a7hQ_iEx0K8ujX1S9ao2cWF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378954.886285_123.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:34.886285208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.005537_124.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:35.005537210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.044075_125.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.044075963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.113951_126.1","section":"bucket","name":"axxsjh-21","timestamp":"2020-03-28 07:02:35.113951241Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hvi1cpckI47MbOR07jxq7tG","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.717034_127.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.717034672Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.837085_128.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:35.837085674Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378955.876631_129.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.876631464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585378955.974300_130.1","section":"bucket","name":"axxsjh-22","timestamp":"2020-03-28 07:02:35.974300064Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_utfuV30GEpqSVZkNmH9cTFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.806273_131.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.806273600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.874519_132.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:36.874519868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378956.904938_133.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.904938372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"write"}}},{"id":"1_1585378956.962527_134.1","section":"bucket","name":"axxsjh-23","timestamp":"2020-03-28 07:02:36.962527960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yzpZFT-5Mgt3oG4DUNuncmC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585378957.415262_135.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.415262594Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.584073_136.1","section":"bucket.instance","name":"axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","timestamp":"2020-03-28 07:02:37.584073810Z","data":{"read_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":1},"write_version":{"tag":"_fUIoJM95AY-ciQrvR4jUSpB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378957.715517_137.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.715517926Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"write"}}},{"id":"1_1585378957.850225_138.1","section":"bucket.instance","name":"axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","timestamp":"2020-03-28 07:02:37.850225621Z","data":{"read_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":1},"write_version":{"tag":"_tr61KxGiUzUy_ppJrkZ3X4S","ver":2},"status":{"status":"complete"}}},{"id":"1_1585378958.196850_139.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.196850880Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"write"}}},{"id":"1_1585378958.514875_140.1","section":"bucket.instance","name":"axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","timestamp":"2020-03-28 07:02:38.514875144Z","data":{"read_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":1},"write_version":{"tag":"_fp2n-C8QHo99gCAXtbEhSDq","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379402.082674_164.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.082674021Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.165724_165.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:02.165724850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.205132_166.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.205132358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.270963_167.1","section":"bucket","name":"axxsjh-24","timestamp":"2020-03-28 07:10:02.270963399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tndbnEBcDYb7E6hODTVVExm","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379402.766427_168.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.766427962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"write"}}},{"id":"1_1585379402.897891_169.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:02.897891401Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379403.013350_170.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.013350905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"write"}}},{"id":"1_1585379403.263728_171.1","section":"bucket","name":"axxsjh-25","timestamp":"2020-03-28 07:10:03.263728640Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8EB2to8WJWeiIBSLhbLycHW","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.150477_172.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.150477693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.217340_173.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:04.217340110Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.252681_174.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.252681155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"write"}}},{"id":"1_1585379404.441937_175.1","section":"bucket","name":"axxsjh-26","timestamp":"2020-03-28 07:10:04.441937707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MR374aKkgAdPNnAJ4CgQyQd","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379404.841841_176.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.841841152Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"write"}}},{"id":"1_1585379404.939836_177.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:04.939836691Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":1},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379405.021631_178.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.021631740Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"write"}}},{"id":"1_1585379405.138798_179.1","section":"bucket.instance","name":"axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","timestamp":"2020-03-28 07:10:05.138798684Z","data":{"read_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":2},"write_version":{"tag":"_xfmmNvbUxQ0WlvX6046YcTa","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379405.372671_180.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.372671815Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"write"}}},{"id":"1_1585379405.906068_181.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:05.906068792Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":1},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379406.212039_182.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.212039589Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"write"}}},{"id":"1_1585379406.487900_183.1","section":"bucket.instance","name":"axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","timestamp":"2020-03-28 07:10:06.487900992Z","data":{"read_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":2},"write_version":{"tag":"_qKvxhLH7vpOl26J50gIWZFw","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379406.872428_184.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:06.872428649Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"write"}}},{"id":"1_1585379407.158264_185.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.158264055Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":1},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379407.405416_186.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.405416952Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"write"}}},{"id":"1_1585379407.585682_187.1","section":"bucket.instance","name":"axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","timestamp":"2020-03-28 07:10:07.585682617Z","data":{"read_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":2},"write_version":{"tag":"_SOcEpQ5Ite0xTwXpishFV3W","ver":3},"status":{"status":"complete"}}},{"id":"1_1585379850.371148_196.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.371148834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.460427_197.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:30.460427247Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.545339_198.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.545339790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"write"}}},{"id":"1_1585379850.668860_199.1","section":"bucket","name":"axxsjh-27","timestamp":"2020-03-28 07:17:30.668860950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vu9ZDYjxt-O8TTaOqLkPBJ4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379850.979253_200.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:30.979253727Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.049602_201.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:31.049602101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.082167_202.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.082167494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379851.167145_203.1","section":"bucket","name":"axxsjh-28","timestamp":"2020-03-28 07:17:31.167145985Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jXvc2nZWpQLdix2XexWqIIJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379851.929991_204.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:31.929991834Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.204743_205.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:32.204743108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.301008_206.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.301008778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379852.388854_207.1","section":"bucket","name":"axxsjh-29","timestamp":"2020-03-28 07:17:32.388854734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bkSDcRHRna5s-U6kjB2vnlu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379852.722625_208.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.722625250Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"write"}}},{"id":"1_1585379852.892466_209.1","section":"bucket.instance","name":"axxsjh-27:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36","timestamp":"2020-03-28 07:17:32.892466771Z","data":{"read_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":1},"write_version":{"tag":"_k1v_tVUV_AZoLsJUdcqlchF","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.190280_210.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.190280998Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"write"}}},{"id":"1_1585379853.451679_211.1","section":"bucket.instance","name":"axxsjh-28:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37","timestamp":"2020-03-28 07:17:33.451679986Z","data":{"read_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":1},"write_version":{"tag":"_6nxDbdIxMTmYj2u5lChT0qv","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379853.819439_212.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:33.819439713Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"write"}}},{"id":"1_1585379854.172629_213.1","section":"bucket.instance","name":"axxsjh-29:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38","timestamp":"2020-03-28 07:17:34.172629545Z","data":{"read_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":1},"write_version":{"tag":"_XHSYrw8fSZkjpuoPQpKytpC","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379854.583256_214.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.583256878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.664872_215.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:34.664872210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379854.740120_216.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.740120131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"write"}}},{"id":"1_1585379854.889105_217.1","section":"bucket","name":"axxsjh-30","timestamp":"2020-03-28 07:17:34.889105146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pcU2GI9Dau1eJo1EA5qeP75","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.647104_218.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.647104317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.731907_219.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:35.731907759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379855.782813_220.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.782813367Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"write"}}},{"id":"1_1585379855.851677_221.1","section":"bucket","name":"axxsjh-31","timestamp":"2020-03-28 07:17:35.851677140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H69yR4mpu1GGsEUaGNq5qWn","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.669353_222.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.669353521Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.742235_223.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:36.742235698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379856.777249_224.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.777249761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379856.840487_225.1","section":"bucket","name":"axxsjh-32","timestamp":"2020-03-28 07:17:36.840487434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4KuBhW0eVPHDN8D0MVx74iu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379857.227131_227.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.227131243Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"write"}}},{"id":"1_1585379857.545595_228.1","section":"bucket.instance","name":"axxsjh-30:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39","timestamp":"2020-03-28 07:17:37.545595233Z","data":{"read_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":1},"write_version":{"tag":"_D3shlL_FBDO9v0wYE_550IB","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379858.131541_229.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.131541166Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"write"}}},{"id":"1_1585379858.424567_230.1","section":"bucket.instance","name":"axxsjh-31:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40","timestamp":"2020-03-28 07:17:38.424567718Z","data":{"read_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":1},"write_version":{"tag":"_1jRTBD7kEdLTAAMvTf2kW20","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379859.203855_231.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.203855216Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"write"}}},{"id":"1_1585379859.381569_232.1","section":"bucket.instance","name":"axxsjh-32:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41","timestamp":"2020-03-28 07:17:39.381569502Z","data":{"read_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":1},"write_version":{"tag":"_yHBgBgJV84Z1q6s2anCqJ9l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379860.007665_234.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.007665223Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.086952_235.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:40.086952164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.122074_236.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.122074070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.186751_237.1","section":"bucket","name":"axxsjh-33","timestamp":"2020-03-28 07:17:40.186751289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoK66BO8-uRyMwq2FnyCenT","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379860.543598_238.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.543598918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"write"}}},{"id":"1_1585379860.904647_239.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:40.904647375Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379861.031309_240.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.031309531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"write"}}},{"id":"1_1585379861.294598_241.1","section":"bucket","name":"axxsjh-34","timestamp":"2020-03-28 07:17:41.294598857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RgfRmOC6VO6dmpNPwfSFmws","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.193860_242.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.193860827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.315249_243.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:42.315249242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.369104_244.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.369104369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"write"}}},{"id":"1_1585379862.439339_245.1","section":"bucket","name":"axxsjh-35","timestamp":"2020-03-28 07:17:42.439339099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VSWYOu2OrcVtcTiI6iErZvF","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379862.787211_246.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:42.787211587Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"write"}}},{"id":"1_1585379863.098274_247.1","section":"bucket.instance","name":"axxsjh-33:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42","timestamp":"2020-03-28 07:17:43.098274295Z","data":{"read_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":1},"write_version":{"tag":"_6tL0CMbRmbM2Wv3TY02A4Hb","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379864.309844_248.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.309844529Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"write"}}},{"id":"1_1585379864.567708_249.1","section":"bucket.instance","name":"axxsjh-34:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43","timestamp":"2020-03-28 07:17:44.567708251Z","data":{"read_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":1},"write_version":{"tag":"_ABq0u-c0UTudFdeS4Gxlm_U","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379865.935373_250.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:45.935373074Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"write"}}},{"id":"1_1585379866.043999_251.1","section":"bucket.instance","name":"axxsjh-35:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44","timestamp":"2020-03-28 07:17:46.043999793Z","data":{"read_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":1},"write_version":{"tag":"_BktrQ7IrrULa0rDHV7cQVOZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585379866.362803_252.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.362803884Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.434580_253.1","section":"bucket.instance","name":"axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","timestamp":"2020-03-28 07:17:46.434580664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sgf2VYeLEjyIITg9m84g9Qu","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379866.484419_254.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.484419326Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"write"}}},{"id":"1_1585379866.563791_255.1","section":"bucket","name":"axxsjh-36","timestamp":"2020-03-28 07:17:46.563791882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X1hIPfDOrcbwBB_BHXA-Zpa","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.020349_256.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.020349101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.312591_257.1","section":"bucket.instance","name":"axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","timestamp":"2020-03-28 07:17:47.312591356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_go8t7Bur7zbBkCA8t-1vKuk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379867.395334_258.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.395334459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379867.497469_259.1","section":"bucket","name":"axxsjh-37","timestamp":"2020-03-28 07:17:47.497469060Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OWEoy_uswhpYYXzcWnTAtRA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.309560_260.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.309560719Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.426071_261.1","section":"bucket.instance","name":"axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","timestamp":"2020-03-28 07:17:48.426071387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3eAZ9p7jcMtZ_2zNWKQ38_3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379868.506294_262.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.506294170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"write"}}},{"id":"1_1585379868.672446_263.1","section":"bucket","name":"axxsjh-38","timestamp":"2020-03-28 07:17:48.672446537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K0MsBrtJOfG50f7q5f5W6WY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.418476_264.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.418476376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.557238_265.1","section":"bucket.instance","name":"axxsjh-39:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48","timestamp":"2020-03-28 07:17:57.557238989Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_54K0pWt2LMtre7DAcqKfBG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379877.646448_266.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.646448712Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"write"}}},{"id":"1_1585379877.943421_267.1","section":"bucket","name":"axxsjh-39","timestamp":"2020-03-28 07:17:57.943421678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5bnPM03CqJCRg_V7iaglKFh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.459199_268.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.459199595Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"write"}}},{"id":"1_1585379878.695408_269.1","section":"bucket.instance","name":"axxsjh-40:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49","timestamp":"2020-03-28 07:17:58.695408966Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A34HxXxJLJpOvPW5g5SDu-c","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379878.956601_270.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:58.956601405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"write"}}},{"id":"1_1585379879.324506_271.1","section":"bucket","name":"axxsjh-40","timestamp":"2020-03-28 07:17:59.324506383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_supP3xDipTh2552riYUPSJp","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379880.595995_272.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:00.595995959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.038833_273.1","section":"bucket.instance","name":"axxsjh-41:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50","timestamp":"2020-03-28 07:18:01.038833931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WCx9ESyKCHekmk9p9pJvltk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379881.172789_274.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.172789120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"write"}}},{"id":"1_1585379881.274012_275.1","section":"bucket","name":"axxsjh-41","timestamp":"2020-03-28 07:18:01.274012862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DiEATdagaxFseZGx7ki0fLX","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376740.861003_77.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:40.861003069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"write"}}},{"id":"1_1585376742.706900_78.1","section":"bucket.instance","name":"axxsjh-1:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1","timestamp":"2020-03-28 06:25:42.706900935Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y_r2y370-yI5LSznMz1C416","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376743.543331_79.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:43.543331546Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"write"}}},{"id":"1_1585376744.020908_80.1","section":"bucket","name":"axxsjh-1","timestamp":"2020-03-28 06:25:44.020908648Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cICVth3H0llIyq8x-JnYqGf","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376749.252320_81.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:49.252320307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.065997_82.1","section":"bucket.instance","name":"axxsjh-2:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2","timestamp":"2020-03-28 06:25:52.065997004Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tvKt5Cn5EeAG3QOEM8ip4d4","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376752.396205_83.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.396205990Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"write"}}},{"id":"1_1585376752.464236_84.1","section":"bucket","name":"axxsjh-2","timestamp":"2020-03-28 06:25:52.464236341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_e_uXZawVgLD96E4kO6pDpPU","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.210946_85.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.210946488Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.347177_86.1","section":"bucket.instance","name":"axxsjh-3:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3","timestamp":"2020-03-28 06:25:56.347177695Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-IShtsqWx4oUENQwdsWFrAh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376756.402195_87.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.402195270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376756.492951_88.1","section":"bucket","name":"axxsjh-3","timestamp":"2020-03-28 06:25:56.492951180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXcXIjgyv3NwRB07Fh2aYW2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.315487_89.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.315487274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.391442_90.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4","timestamp":"2020-03-28 06:26:03.391442919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QMlnMjj47ZuH6RHs233u_mo","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376763.457622_91.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.457622052Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"write"}}},{"id":"1_1585376763.551172_92.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:03.551172358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xC1ZlCdY2wdvPiL7OCaLz-E","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.182472_93.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.182472572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.357825_94.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5","timestamp":"2020-03-28 06:26:04.357825619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_miNRe_lT_boTsx1dQCEf0NE","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376764.511973_95.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.511973399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"write"}}},{"id":"1_1585376764.667779_96.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:04.667779763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cXptPLrgoCdLuavjO_uSvgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.340138_97.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.340138823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.454915_98.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6","timestamp":"2020-03-28 06:26:05.454915090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iuKmFkPrSg1rPgbFbFKWRU2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376765.494752_99.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.494752076Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"write"}}},{"id":"1_1585376765.686952_100.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:05.686952911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GbvLAnmKUa8IOdOXztuG4px","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.580122_101.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.580122454Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.659843_102.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.659843259Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376772.689935_103.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.689935725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"write"}}},{"id":"1_1585376772.763938_104.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:12.763938293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fcVRPiI0xtC9bIE9MMujclm","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376772.846391_105.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.846391689Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376772.953845_106.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.7","timestamp":"2020-03-28 06:26:12.953845611Z","data":{"read_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":1},"write_version":{"tag":"_9XrzhA37_Yv0yZ1LMT1yD5l","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.145320_107.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.145320320Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.206662_108.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.206662772Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.237606_109.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.237606068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.291875_110.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:13.291875555Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8tA0zr_eiF8045m4S2pG8mS","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376773.349896_111.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.349896163Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376773.492815_112.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.8","timestamp":"2020-03-28 06:26:13.492815556Z","data":{"read_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":1},"write_version":{"tag":"_r8fzo0T4jZwALFe2ocjDE2z","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376773.864643_113.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.864643930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376773.940990_114.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:13.940990102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376773.976369_115.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:13.976369166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.101523_116.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:14.101523605Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ahw8HNrHSifQZdreUzqwJnB","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376774.228124_117.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.228124246Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376774.340781_118.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.9","timestamp":"2020-03-28 06:26:14.340781699Z","data":{"read_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":1},"write_version":{"tag":"_XrImmjinxxADfSswWMU_sh2","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376774.665269_119.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.665269386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.731256_120.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:14.731256315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376774.812674_121.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.812674535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"write"}}},{"id":"1_1585376774.896072_122.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:14.896072588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CGlV2J-NIljCYzEgnKFr7i8","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376775.027616_123.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.027616069Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376775.260905_124.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.10","timestamp":"2020-03-28 06:26:15.260905974Z","data":{"read_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":1},"write_version":{"tag":"_JC3SzbRLvqKtKutu1_KPwRD","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376775.927416_125.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:15.927416490Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.150255_126.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.150255103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376776.204310_127.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.204310139Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"write"}}},{"id":"1_1585376776.283073_128.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:16.283073483Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JSkpbT6aTzHvXaQ4B3fZdc2","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376776.345243_129.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.345243656Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376776.442968_130.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.11","timestamp":"2020-03-28 06:26:16.442968228Z","data":{"read_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":1},"write_version":{"tag":"_4pptRo-JzK98CmOyj52X4pM","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376777.171070_131.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.171070476Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.251548_132.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.251548808Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376777.313435_133.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.313435517Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"write"}}},{"id":"1_1585376777.473114_134.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:17.473114274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WE7X5AfQigvvBZ-ptBZtGsD","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376777.546664_135.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.546664324Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376777.648190_136.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.12","timestamp":"2020-03-28 06:26:17.648190953Z","data":{"read_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":1},"write_version":{"tag":"_b5DYnxeqV8vZLIUZk5mfIFg","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376778.457614_137.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.457614242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.530866_138.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.530866422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376778.596300_139.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.596300056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"write"}}},{"id":"1_1585376778.723354_140.1","section":"bucket","name":"axxsjh-4","timestamp":"2020-03-28 06:26:18.723354680Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2XtZYfplNrjoBaaJd8YZ1Qz","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376778.889445_141.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.889445931Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376778.993727_142.1","section":"bucket.instance","name":"axxsjh-4:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.13","timestamp":"2020-03-28 06:26:18.993727222Z","data":{"read_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":1},"write_version":{"tag":"_zbEbSD2W3rfDuLUck_IybaZ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376779.554126_143.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.554126870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.615566_144.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.615566263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376779.648136_145.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.648136892Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"write"}}},{"id":"1_1585376779.700575_146.1","section":"bucket","name":"axxsjh-5","timestamp":"2020-03-28 06:26:19.700575070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EbumHIyfr4yW-REdHCXgS4u","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376779.755698_147.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.755698282Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376779.836329_148.1","section":"bucket.instance","name":"axxsjh-5:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.14","timestamp":"2020-03-28 06:26:19.836329369Z","data":{"read_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":1},"write_version":{"tag":"_7DRLkaeOIjmH52nhlBRqkev","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376780.412105_149.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.412105520Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.497832_150.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.497832329Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376780.528435_151.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.528435294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"write"}}},{"id":"1_1585376780.580072_152.1","section":"bucket","name":"axxsjh-6","timestamp":"2020-03-28 06:26:20.580072140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MQurX77h-aeNRlar_ZLCXg6","ver":1},"status":{"status":"abort"}}},{"id":"1_1585376780.637374_153.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.637374040Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376780.715973_154.1","section":"bucket.instance","name":"axxsjh-6:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.15","timestamp":"2020-03-28 06:26:20.715973688Z","data":{"read_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":1},"write_version":{"tag":"_PyUW1GTfVVyXq8DXJccjspJ","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376788.749165_156.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.749165145Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"write"}}},{"id":"1_1585376788.820425_157.1","section":"bucket.instance","name":"axxsjh-7:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16","timestamp":"2020-03-28 06:26:28.820425744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ki6-Zuho4t0zYow6Qjr3t8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376788.893305_158.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:28.893305113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.034890_159.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:29.034890598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.439220_160.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.439220137Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.703446_161.1","section":"bucket.instance","name":"axxsjh-8:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17","timestamp":"2020-03-28 06:26:29.703446147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0US60lxbhYjv0hRPfBsafqk","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376789.751465_162.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.751465528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"write"}}},{"id":"1_1585376789.844685_163.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:29.844685962Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.493616_164.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.493616022Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.560835_165.1","section":"bucket.instance","name":"axxsjh-9:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18","timestamp":"2020-03-28 06:26:30.560835206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqHrkIBKdvZVcxOiiAj0cNR","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376790.595668_166.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.595668729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"write"}}},{"id":"1_1585376790.668987_167.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:30.668987885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"status":{"status":"complete"}}},{"id":"1_1585376797.170139_168.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.170139053Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.255629_169.1","section":"bucket","name":"axxsjh-7","timestamp":"2020-03-28 06:26:37.255629310Z","data":{"read_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":1},"write_version":{"tag":"_51dDeD7-XDzXC1eMSlzFyJ3","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376797.661171_170.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.661171532Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376797.740373_171.1","section":"bucket","name":"axxsjh-8","timestamp":"2020-03-28 06:26:37.740373426Z","data":{"read_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":1},"write_version":{"tag":"_3AjGJXBcRFz_ofxjAL2QyJl","ver":2},"status":{"status":"complete"}}},{"id":"1_1585376798.256348_172.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.256348601Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"remove"}}},{"id":"1_1585376798.437614_173.1","section":"bucket","name":"axxsjh-9","timestamp":"2020-03-28 06:26:38.437614170Z","data":{"read_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":1},"write_version":{"tag":"_37dNAY1A3SUyTupG1zJ9pFi","ver":2},"status":{"status":"complete"}}}] 2020-03-28T07:21:12.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:12.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:12.994 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:14.410 INFO:teuthology.orchestra.run.smithi156.stdout:[{"id":"1_1585379956.845079_91.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.845079347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"write"}}},{"id":"1_1585379956.944258_92.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.944258930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379956.976146_93.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:16.976146603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.042035_94.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:17.042035130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.669020_95.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.669020296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.784041_96.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.784041485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.832519_97.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.832519102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.915346_98.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.915346318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:14.411 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:14.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:14.412 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:15.764 INFO:teuthology.orchestra.run.smithi168.stdout:[{"id":"1_1585379956.845079_91.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.845079347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"write"}}},{"id":"1_1585379956.944258_92.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.944258930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379956.976146_93.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:16.976146603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.042035_94.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:17.042035130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.669020_95.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.669020296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.784041_96.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.784041485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.832519_97.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.832519102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.915346_98.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.915346318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:15.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:15.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:15.765 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:17.041 INFO:teuthology.orchestra.run.smithi156.stdout:[{"id":"1_1585379956.845079_91.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.845079347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"write"}}},{"id":"1_1585379956.944258_92.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.944258930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379956.976146_93.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:16.976146603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.042035_94.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:17.042035130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.669020_95.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.669020296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.784041_96.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.784041485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.832519_97.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.832519102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.915346_98.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.915346318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:17.042 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:17.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:17.042 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:18.444 INFO:teuthology.orchestra.run.smithi168.stdout:[{"id":"1_1585379956.845079_91.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.845079347Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"write"}}},{"id":"1_1585379956.944258_92.1","section":"bucket.instance","name":"axxsjh-42:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1","timestamp":"2020-03-28 07:19:16.944258930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LJdoFtMlIWnecBOgqYXeB62","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379956.976146_93.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:16.976146603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.042035_94.1","section":"bucket","name":"axxsjh-42","timestamp":"2020-03-28 07:19:17.042035130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v_Dm1_We9sjsPNG2vYCt6KA","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.669020_95.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.669020296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.784041_96.1","section":"bucket.instance","name":"axxsjh-43:2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2","timestamp":"2020-03-28 07:19:17.784041485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G80Zgdb-ZPC3BhdScTyLJkB","ver":1},"status":{"status":"complete"}}},{"id":"1_1585379957.832519_97.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.832519102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"write"}}},{"id":"1_1585379957.915346_98.1","section":"bucket","name":"axxsjh-43","timestamp":"2020-03-28 07:19:17.915346318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K1lHpqL8VOWviuLhBPjZ2Ag","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:18.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:18.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:18.445 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:19.759 INFO:teuthology.orchestra.run.smithi156.stdout:[{"id":"1_1585380023.659910_55.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.659910531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.759143_56.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.759143785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380023.801796_57.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.801796794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.883256_58.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.883256215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.422731_59.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.422731731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.543809_60.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.543809226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.577876_61.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.577876470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.645876_62.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.645876216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:19.759 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:19.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:19.760 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:21.151 INFO:teuthology.orchestra.run.smithi168.stdout:[{"id":"1_1585380023.659910_55.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.659910531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.759143_56.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.759143785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380023.801796_57.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.801796794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.883256_58.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.883256215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.422731_59.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.422731731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.543809_60.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.543809226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.577876_61.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.577876470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.645876_62.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.645876216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:21.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:21.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:21.152 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:22.572 INFO:teuthology.orchestra.run.smithi156.stdout:[{"id":"1_1585380023.659910_55.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.659910531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.759143_56.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.759143785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380023.801796_57.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.801796794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.883256_58.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.883256215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.422731_59.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.422731731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.543809_60.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.543809226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.577876_61.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.577876470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.645876_62.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.645876216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:22.573 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:22.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:22.573 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:24.151 INFO:teuthology.orchestra.run.smithi168.stdout:[{"id":"1_1585380023.659910_55.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.659910531Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.759143_56.1","section":"bucket.instance","name":"axxsjh-44:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1","timestamp":"2020-03-28 07:20:23.759143785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T7XgeTdpKQ9LCkihh0W9Udh","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380023.801796_57.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.801796794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"write"}}},{"id":"1_1585380023.883256_58.1","section":"bucket","name":"axxsjh-44","timestamp":"2020-03-28 07:20:23.883256215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B_aRy3vYDPZ8iA3Fgdn6FpY","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.422731_59.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.422731731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.543809_60.1","section":"bucket.instance","name":"axxsjh-45:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2","timestamp":"2020-03-28 07:20:24.543809226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_13zJGbIAvSZf9srqVv0BE4n","ver":1},"status":{"status":"complete"}}},{"id":"1_1585380024.577876_61.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.577876470Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"write"}}},{"id":"1_1585380024.645876_62.1","section":"bucket","name":"axxsjh-45","timestamp":"2020-03-28 07:20:24.645876216Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3gVyVa1r8MdJpje1kbiTsyy","ver":1},"status":{"status":"complete"}}}] 2020-03-28T07:21:24.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:24.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:24.152 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:21:26.272 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:26.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:26.272 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:21:28.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:28.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:28.966 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:21:30.784 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:30.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:21:30.785 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:21:32.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:32.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:32.741 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:34.269 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T07:21:34.270 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:34.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:34.271 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:35.692 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T07:21:35.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:35.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:35.693 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:37.037 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T07:21:37.038 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:37.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'3894ea80-31ba-4896-8e09-e661b246b356', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:37.038 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 3894ea80-31ba-4896-8e09-e661b246b356 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:38.566 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T07:21:38.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:38.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:38.567 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:39.838 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T07:21:39.839 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:39.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:39.839 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:41.160 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T07:21:41.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:41.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:41.160 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:42.437 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T07:21:42.437 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:42.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'571a1648-7110-4ecb-afe2-66045679be41', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:42.438 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 571a1648-7110-4ecb-afe2-66045679be41 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:43.834 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T07:21:43.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:43.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:43.835 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:45.352 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T07:21:45.352 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:45.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:45.353 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:46.644 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T07:21:46.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:46.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:46.645 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:48.093 INFO:teuthology.orchestra.run.smithi156.stdout:[] 2020-03-28T07:21:48.094 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:48.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'mdlog', 'list', '--period', u'0d07b3db-af62-4c83-b2b9-6aa7b82de92e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:48.094 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 mdlog list --period 0d07b3db-af62-4c83-b2b9-6aa7b82de92e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:49.455 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-03-28T07:21:49.456 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-03-28T07:21:49.456 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-46 2020-03-28T07:21:49.988 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-47 2020-03-28T07:21:50.937 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-48 2020-03-28T07:21:53.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:53.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:53.906 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:55.750 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380112.616832_115.1","last_update":"2020-03-28 07:21:52.616832Z"}] 2020-03-28T07:21:55.750 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-03-28T07:21:55.751 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:55.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:55.751 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:57.840 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380115.419003_118.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:55.419003Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:21:57.840 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-03-28T07:21:57.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:57.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:57.841 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:21:59.465 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380116.057071_119.1","last_update":"2020-03-28 07:21:56.057071Z"}] 2020-03-28T07:21:59.466 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-03-28T07:21:59.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:59.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:21:59.467 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:00.714 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5750405467414497023},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380116.057071_119.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:56.057071Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:00.715 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-03-28T07:22:00.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:00.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:00.715 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:01.795 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380116.057071_119.1","last_update":"2020-03-28 07:21:56.057071Z"}] 2020-03-28T07:22:01.795 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-03-28T07:22:01.795 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:01.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:01.796 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:03.833 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3889226293267946400},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380116.057071_119.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:56.057071Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:03.834 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-03-28T07:22:03.834 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:03.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:03.835 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:05.258 INFO:teuthology.orchestra.run.smithi168.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380116.815140_115.1","last_update":"2020-03-28 07:21:56.815140Z"}] 2020-03-28T07:22:05.259 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-03-28T07:22:05.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:05.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:05.259 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:06.710 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":16017607255566841915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380116.815140_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:56.815140Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:06.711 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-03-28T07:22:06.711 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:06.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:06.711 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:08.048 INFO:teuthology.orchestra.run.smithi168.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380116.815140_115.1","last_update":"2020-03-28 07:21:56.815140Z"}] 2020-03-28T07:22:08.049 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-03-28T07:22:08.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:08.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:08.050 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:09.677 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2416084121408814850},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380116.815140_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:56.815140Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:09.678 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-03-28T07:22:09.678 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:09.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:09.679 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:11.060 INFO:teuthology.orchestra.run.smithi168.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380116.815140_115.1","last_update":"2020-03-28 07:21:56.815140Z"}] 2020-03-28T07:22:11.061 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone2 2020-03-28T07:22:11.061 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:11.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:11.062 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:12.562 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:12.563 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:22:17.563 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:17.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:17.564 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:19.549 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:19.549 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:22:24.551 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:24.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:24.551 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:26.495 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:26.496 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:22:31.497 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:31.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:31.497 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:33.337 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:33.338 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:22:38.338 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:38.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:38.339 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:40.078 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:40.078 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:22:45.079 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:45.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:45.080 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:47.210 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:47.211 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:22:52.213 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:52.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:52.213 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:22:54.207 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:22:54.208 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:22:59.209 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:59.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:22:59.209 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:01.433 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:01.434 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:05.269 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:05:37.239 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:23:05.273 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:05:37.244 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:23:05.613 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:05:37.583 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:23:06.435 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:06.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:06.435 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:08.245 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:08.245 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:13.246 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:13.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:13.247 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:15.221 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:15.221 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:20.222 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:20.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:20.224 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:22.122 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:22.123 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:27.124 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:27.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:27.125 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:29.164 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:29.165 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:34.165 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:34.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:34.166 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:36.154 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:36.154 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:41.155 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:41.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:41.156 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:43.448 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:43.449 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:48.449 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:48.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:48.450 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:50.390 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:50.391 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:23:55.391 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:55.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:23:55.392 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:23:57.465 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:23:57.466 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:02.467 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:02.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:02.467 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:04.572 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:04.573 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:09.573 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:09.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:09.574 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:11.653 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:11.653 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:16.654 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:16.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:16.655 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:18.543 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:18.543 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:23.288 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:05:49.349 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:24:23.309 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:05:49.370 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:24:23.544 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:23.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:23.545 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:23.640 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:05:49.701 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:24:26.282 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:26.283 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:31.283 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:31.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:31.284 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:33.424 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:33.424 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:38.425 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:38.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:38.426 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:40.588 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:40.589 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:45.589 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:45.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:45.590 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:47.476 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:47.477 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:51.708 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:07:23.679 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:24:51.710 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:07:23.681 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:24:51.791 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:07:23.761 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:24:52.477 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:52.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:52.478 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:24:54.359 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:24:54.360 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:24:59.360 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:59.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:24:59.361 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:01.644 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:01.644 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:06.645 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:06.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:06.646 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:08.578 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:08.578 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:13.579 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:13.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:13.580 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:15.449 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:15.450 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:20.450 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:20.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:20.451 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:22.452 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:22.453 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:27.454 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:27.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:27.454 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:29.524 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:29.525 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:34.526 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:34.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:34.526 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:36.473 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:36.473 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:41.474 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:41.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:41.475 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:43.651 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:43.651 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:48.652 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:48.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:48.653 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:50.675 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:50.675 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:25:55.676 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:55.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:25:55.677 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:25:57.349 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:07:23.410 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:25:57.352 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:07:23.413 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:25:57.467 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:07:23.529 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:25:57.908 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:25:57.908 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:02.908 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:02.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:02.909 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:04.894 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:04.895 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:09.895 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:09.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:09.896 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:11.929 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:11.938 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:16.939 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:16.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:16.940 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:18.878 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:18.878 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:23.879 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:23.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:23.879 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:25.798 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:25.798 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:30.798 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:30.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:30.799 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:32.908 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:32.909 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:37.910 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:37.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:37.911 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:39.647 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:39.648 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:44.649 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:44.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:44.650 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:46.716 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:46.717 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:51.718 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:51.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:51.718 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:26:53.930 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:26:53.930 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:26:58.931 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:58.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:26:58.931 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:00.854 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:00.855 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:05.856 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:05.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:05.856 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:07.784 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:07.784 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:12.785 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:12.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:12.785 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:14.676 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:14.677 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:19.677 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:19.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:19.678 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:21.612 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:21.612 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:26.614 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:26.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:26.615 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:28.613 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:28.614 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:33.615 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:33.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:33.615 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:35.653 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:35.653 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:40.654 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:40.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:40.655 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:42.531 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:42.531 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:47.532 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:47.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:47.532 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:49.676 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:49.676 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:27:54.677 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:54.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:27:54.677 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:27:56.601 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:27:56.602 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:01.602 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:01.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:01.603 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:03.674 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:03.675 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:08.676 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:08.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:08.677 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:10.805 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:10.805 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:15.806 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:15.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:15.806 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:17.707 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:17.708 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:22.709 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:22.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:22.710 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:24.937 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:24.938 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:29.939 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:29.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:29.939 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:32.136 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:32.136 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:37.137 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:37.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:37.138 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:39.113 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:39.113 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:44.114 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:44.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:44.114 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:46.083 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:46.084 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:51.085 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:51.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:51.085 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:28:52.964 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:28:52.964 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:28:57.965 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:57.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:28:57.966 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:29:00.271 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:29:00.271 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:29:05.272 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:29:05.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:29:05.272 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:29:07.327 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:29:07.328 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:29:12.340 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... FAIL 2020-03-28T07:29:12.340 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:29:12.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:29:12.341 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:29:12.788 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:29:12.785 7f4a52241840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T07:29:12.788 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:29:12.786 7f4a52241840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T07:29:12.793 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:29:12.791 7f4a52241840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T07:29:12.793 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:29:12.791 7f4a52241840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T07:29:12.871 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:29:12.871 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:29:12.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:29:12.872 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:29:14.101 INFO:teuthology.orchestra.run.smithi168.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T07:29:16.854 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":2,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:29:16.855 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-03-28T07:29:59.847 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:12:31.817 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:29:59.855 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:12:31.825 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:30:00.155 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:12:32.125 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:30:16.856 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-49 2020-03-28T07:30:17.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:17.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:17.741 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:18.669 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T07:30:18.961 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585380617.427131_85.1","last_update":"2020-03-28 07:30:17.427131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:30:18.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:30:18.962 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:18.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:18.962 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:20.462 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380111.997572_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:51.997572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:20.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1585380617.427131_85.1 target=1_1585380111.997572_77.1 2020-03-28T07:30:25.464 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:25.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:25.465 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:26.864 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380111.997572_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:51.997572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:26.864 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1585380617.427131_85.1 target=1_1585380111.997572_77.1 2020-03-28T07:30:31.865 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:31.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:31.866 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:33.253 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380111.997572_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:51.997572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:33.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1585380617.427131_85.1 target=1_1585380111.997572_77.1 2020-03-28T07:30:38.254 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:38.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:38.255 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:39.657 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380111.997572_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:51.997572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:39.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1585380617.427131_85.1 target=1_1585380111.997572_77.1 2020-03-28T07:30:44.658 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:44.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:44.658 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:46.176 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380111.997572_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:21:51.997572Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:46.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1585380617.427131_85.1 target=1_1585380111.997572_77.1 2020-03-28T07:30:51.178 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:51.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:51.179 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:52.583 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380617.427131_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:30:17.427131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:52.584 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:30:52.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:30:52.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:52.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:52.585 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:54.089 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380617.427131_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:30:17.427131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:54.090 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:30:54.090 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:30:54.090 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:54.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:30:54.091 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:30:56.205 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380617.427131_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:30:17.427131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:30:56.206 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:30:56.454 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5fb75e0a-cb77-479b-b462-3a1befb8717c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:30:56.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=5fb75e0a-cb77-479b-b462-3a1befb8717c', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:30:56.455 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=5fb75e0a-cb77-479b-b462-3a1befb8717c --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:30:56.837 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:30:56.834 7f8820848840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T07:30:56.837 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:30:56.834 7f8820848840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T07:30:56.842 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:30:56.839 7f8820848840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T07:30:56.842 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:30:56.839 7f8820848840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T07:30:56.914 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:30:56.914 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:30:56.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:30:56.915 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:30:58.364 INFO:teuthology.orchestra.run.smithi168.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T07:31:00.801 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":3,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c"},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:31:00.802 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "5fb75e0a-cb77-479b-b462-3a1befb8717c" 2020-03-28T07:31:17.952 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:12:44.013 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:31:17.955 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:12:44.017 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:31:18.172 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:12:44.233 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:32:03.280 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:32:03.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:32:03.281 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:32:03.633 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:32:03.631 7f686e995840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T07:32:03.633 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:32:03.631 7f686e995840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T07:32:03.637 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:32:03.636 7f686e995840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T07:32:03.637 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:32:03.636 7f686e995840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T07:32:03.719 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:32:03.719 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:32:03.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:32:03.720 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:32:05.092 INFO:teuthology.orchestra.run.smithi168.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T07:32:08.559 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":4,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c"},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:32:08.560 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-03-28T07:33:08.560 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:33:08.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:33:08.561 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:33:09.016 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:33:09.013 7fc3dbbaf840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T07:33:09.016 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:33:09.013 7fc3dbbaf840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T07:33:09.021 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:33:09.019 7fc3dbbaf840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T07:33:09.022 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:33:09.019 7fc3dbbaf840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T07:33:09.103 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:33:09.104 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:33:09.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:33:09.104 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:33:10.610 INFO:teuthology.orchestra.run.smithi168.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T07:33:12.370 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:15:44.340 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:33:12.396 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:15:44.366 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:33:12.485 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:15:44.455 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:33:13.750 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":5,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:33:13.750 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-03-28T07:34:13.751 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-03-28T07:34:13.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:13.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:13.752 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:34:14.977 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T07:34:15.285 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585380617.427131_85.1","last_update":"2020-03-28 07:30:17.427131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:34:15.286 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:34:15.286 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:15.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:15.286 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:34:16.759 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380617.427131_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:30:17.427131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:34:16.759 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:34:16.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:34:16.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:16.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:16.760 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:34:18.104 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380617.427131_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:30:17.427131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:34:18.105 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:34:18.105 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:34:18.105 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:18.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:34:18.106 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:34:18.213 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:15:44.274 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:34:18.216 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:15:44.277 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:34:18.443 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:15:44.504 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:34:20.483 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585380617.427131_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:30:17.427131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:34:20.483 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:34:20.483 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', u'BQ8OGVQAZXH9K7HK', '--secret', u'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:20.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', u'BQ8OGVQAZXH9K7HK', '--secret', u'eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:20.484 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key BQ8OGVQAZXH9K7HK --secret eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:34:21.195 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:21.192 7f2ec8894840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T07:34:21.195 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:21.192 7f2ec8894840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T07:34:21.200 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:21.197 7f2ec8894840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T07:34:21.200 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:21.197 7f2ec8894840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T07:34:21.238 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"1e2b7e2b-c91d-4dec-b80b-9fb1043f036c","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"BQ8OGVQAZXH9K7HK","secret_key":"eFrb0YkllUpa2QL7W38b3Bi7BaV32OU7"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:34:21.239 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['period', 'update', '--rgw-zone', u'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:21.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'period', 'update', '--rgw-zone', u'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:21.239 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:34:52.432 INFO:teuthology.orchestra.run.smithi168.stdout:Sending period to new master zone 5fb75e0a-cb77-479b-b462-3a1befb8717c 2020-03-28T07:34:55.040 INFO:teuthology.orchestra.run.smithi168.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":6,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"1e2b7e2b-c91d-4dec-b80b-9fb1043f036c","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"1e2b7e2b-c91d-4dec-b80b-9fb1043f036c","val":294856592},{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:34:55.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', u'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:55.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', u'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:55.041 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:34:55.341 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:34:55.338 7fe91af38840 0 failed reading obj info from .rgw.root:zone_info.2c0a3e73-a280-4ccf-960d-d2c86331b6ff: (2) No such file or directory 2020-03-28T07:34:55.341 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:34:55.338 7fe91af38840 0 WARNING: could not read zone params for zone id=2c0a3e73-a280-4ccf-960d-d2c86331b6ff name=test-zone2 2020-03-28T07:34:55.361 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:34:55.359 7fe91af38840 0 failed reading obj info from .rgw.root:zone_info.d8382bd8-febb-4ae8-8d2a-8fafdd38eff5: (2) No such file or directory 2020-03-28T07:34:55.362 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:34:55.359 7fe91af38840 0 WARNING: could not read zone params for zone id=d8382bd8-febb-4ae8-8d2a-8fafdd38eff5 name=test-zone4 2020-03-28T07:34:55.426 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:34:55.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', u'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:55.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', u'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:55.427 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:34:55.659 INFO:teuthology.orchestra.run.smithi156.stderr:zone name remove not found in zonegroup test-zonegroup 2020-03-28T07:34:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-03-28T07:34:55.664 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'delete', '--rgw-zone', u'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:55.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', u'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:55.665 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-03-28T07:34:56.574 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:56.571 7f26876b6840 0 failed reading obj info from .rgw.root:zone_info.5fb75e0a-cb77-479b-b462-3a1befb8717c: (2) No such file or directory 2020-03-28T07:34:56.574 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:56.571 7f26876b6840 0 WARNING: could not read zone params for zone id=5fb75e0a-cb77-479b-b462-3a1befb8717c name=test-zone1 2020-03-28T07:34:56.582 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:56.579 7f26876b6840 0 failed reading obj info from .rgw.root:zone_info.85494750-2c5c-47db-a993-372e454c1eb9: (2) No such file or directory 2020-03-28T07:34:56.582 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:56.579 7f26876b6840 0 WARNING: could not read zone params for zone id=85494750-2c5c-47db-a993-372e454c1eb9 name=test-zone3 2020-03-28T07:34:56.652 INFO:teuthology.orchestra.run.smithi168.stderr:2020-03-28 07:34:56.649 7f26876b6840 0 zone id 1e2b7e2b-c91d-4dec-b80b-9fb1043f036c is not a part of zonegroup default 2020-03-28T07:34:56.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:56.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:34:56.726 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:34:57.693 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:34:57.690 7f5b95af6840 0 WARNING: can't generate connection for zone 1e2b7e2b-c91d-4dec-b80b-9fb1043f036c id remove: no endpoints defined 2020-03-28T07:35:00.904 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":7,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:35:00.905 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-03-28T07:35:00.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:00.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:00.906 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:01.409 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"0a1374fa-b0e1-4242-921e-8a9b71d97b92","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:35:01.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', u'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:01.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', u'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:01.410 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:02.304 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"cb53091b-a35d-4814-ba22-a2f2d500eeb1","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"metadata_heap":"","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"} 2020-03-28T07:35:02.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:02.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:02.305 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:05.475 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":8,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"0a1374fa-b0e1-4242-921e-8a9b71d97b92","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"cb53091b-a35d-4814-ba22-a2f2d500eeb1","zones":[{"id":"cb53091b-a35d-4814-ba22-a2f2d500eeb1","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"cb53091b-a35d-4814-ba22-a2f2d500eeb1","val":582287773},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:35:05.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', u'remove', '--rgw-zonegroup', u'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:05.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', u'remove', '--rgw-zonegroup', u'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:05.476 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:06.180 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:35:06.177 7fa2b9a0c840 0 zone id cb53091b-a35d-4814-ba22-a2f2d500eeb1 is not a part of zonegroup test-zonegroup 2020-03-28T07:35:06.219 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:35:06.216 7fa2b9a0c840 0 zone id cb53091b-a35d-4814-ba22-a2f2d500eeb1 is not a part of zonegroup default 2020-03-28T07:35:06.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:06.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:06.335 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:06.968 INFO:teuthology.orchestra.run.smithi156.stderr:2020-03-28 07:35:06.965 7fc5c9441840 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-03-28T07:35:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-03-28T07:35:06.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', u'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:06.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', u'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:06.972 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:07.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:07.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:07.312 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:10.043 INFO:teuthology.orchestra.run.smithi156.stdout:{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","epoch":9,"predecessor_uuid":"571a1648-7110-4ecb-afe2-66045679be41","sync_status":["1_1585379957.915346_98.1","","",""],"period_map":{"id":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","zones":[{"id":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","name":"test-zone2","endpoints":["http://smithi168.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"5fb75e0a-cb77-479b-b462-3a1befb8717c","name":"test-zone1","endpoints":["http://smithi156.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"85494750-2c5c-47db-a993-372e454c1eb9","name":"test-zone3","endpoints":["http://smithi156.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","name":"test-zone4","endpoints":["http://smithi168.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":0,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc"}],"short_zone_ids":[{"key":"2c0a3e73-a280-4ccf-960d-d2c86331b6ff","val":1061870527},{"key":"5fb75e0a-cb77-479b-b462-3a1befb8717c","val":383014882},{"key":"85494750-2c5c-47db-a993-372e454c1eb9","val":4171428995},{"key":"d8382bd8-febb-4ae8-8d2a-8fafdd38eff5","val":1985772535}]},"master_zonegroup":"test-zonegroup","master_zone":"5fb75e0a-cb77-479b-b462-3a1befb8717c","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"b24b1e69-ff48-4356-b1b9-8937c2aaecdc","realm_name":"test-realm","realm_epoch":4} 2020-03-28T07:35:10.043 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-03-28T07:35:10.044 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-50 2020-03-28T07:35:14.299 INFO:tasks.rgw.c1.client.1.smithi156.stdout:2020-03-28 07:35:14.285 776e6700 -1 failed to decode period 2020-03-28T07:35:15.166 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-51 2020-03-28T07:35:16.473 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-52 2020-03-28T07:35:17.887 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-53 2020-03-28T07:35:18.728 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-54 2020-03-28T07:35:32.332 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-55 2020-03-28T07:35:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-03-28T07:35:36.797 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-56 2020-03-28T07:35:37.270 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-57 2020-03-28T07:35:38.446 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-58 2020-03-28T07:35:39.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'axxsjh-56', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:39.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'axxsjh-56', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:39.302 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket axxsjh-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:40.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'axxsjh-57', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:40.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'axxsjh-57', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:40.810 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket axxsjh-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:42.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'axxsjh-58', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:42.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'axxsjh-58', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-03-28T07:35:42.206 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket axxsjh-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-03-28T07:35:43.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', u'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:43.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', u'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:43.606 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:35:44.925 INFO:teuthology.orchestra.run.smithi156.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17-axxsjh-8","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","obj_delete_at_hint.0000000028","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27-axxsjh-18","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25-axxsjh-16","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","obj_delete_at_hint.0000000015","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3-axxsjh-3","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40-axxsjh-31","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1-axxsjh-49","obj_delete_at_hint.0000000047","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.3","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.0.retry","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.2","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","datalog.sync-status.85494750-2c5c-47db-a993-372e454c1eb9","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24-axxsjh-15","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2-axxsjh-45","2020-03-28-07-2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1-axxsjh-42","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.1","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.2","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.8-axxsjh-57","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2-axxsjh-2","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29-axxsjh-20","obj_delete_at_hint.0000000052","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4-axxsjh-47","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","obj_delete_at_hint.0000000096","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.3.retry","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-49:5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50-axxsjh-41","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49-axxsjh-40","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38-axxsjh-29","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48-axxsjh-39","obj_delete_at_hint.0000000064","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33-axxsjh-24","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37-axxsjh-28","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-46:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43-axxsjh-34","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28-axxsjh-19","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.4-axxsjh-53","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.6-axxsjh-55","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.0","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26-axxsjh-17","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-46:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21-axxsjh-12","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","mdlog.sync-status.shard.2","2020-03-28-07-2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2-axxsjh-43","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23-axxsjh-14","obj_delete_at_hint.0000000116","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45-axxsjh-36","obj_delete_at_hint.0000000034","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.1","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-49:5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6-axxsjh-6","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30-axxsjh-21","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-48:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5","obj_delete_at_hint.0000000073","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16-axxsjh-7","obj_delete_at_hint.0000000038","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.2","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32-axxsjh-23","meta.full-sync.index.3","obj_delete_at_hint.0000000118","datalog.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41-axxsjh-32","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.0","obj_delete_at_hint.0000000119","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","obj_delete_at_hint.0000000098","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.2-axxsjh-51","mdlog.sync-status.shard.0","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","obj_delete_at_hint.0000000027","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35-axxsjh-26","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18-axxsjh-9","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1-axxsjh-44","obj_delete_at_hint.0000000039","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5-axxsjh-48","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.3-axxsjh-52","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","obj_delete_at_hint.0000000100","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.5-axxsjh-54","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5-axxsjh-5","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.39-axxsjh-30","obj_delete_at_hint.0000000057","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.42-axxsjh-33","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","obj_delete_at_hint.0000000012","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-48:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.2","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3-axxsjh-46","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22-axxsjh-13","obj_delete_at_hint.0000000025","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.36-axxsjh-27","obj_delete_at_hint.0000000123","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.3","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.7-axxsjh-56","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","obj_delete_at_hint.0000000013","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.1","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44-axxsjh-35","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31-axxsjh-22","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-47:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.0","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34-axxsjh-25","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-47:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.3","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4-axxsjh-4","meta.full-sync.index.2","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20-axxsjh-11","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.9-axxsjh-58","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.3.retry","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.0","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","meta.log.0d07b3db-af62-4c83-b2b9-6aa7b82de92e.0","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.3","obj_delete_at_hint.0000000097","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","obj_delete_at_hint.0000000042","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.1-axxsjh-50","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19-axxsjh-10","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47-axxsjh-38","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1-axxsjh-1","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46-axxsjh-37"] 2020-03-28T07:35:45.199 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['log', 'list', '--rgw-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:45.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'log', 'list', '--rgw-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:45.200 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:35:46.949 INFO:teuthology.orchestra.run.smithi168.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17-axxsjh-8","obj_delete_at_hint.0000000028","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25-axxsjh-16","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-47:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","obj_delete_at_hint.0000000015","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3-axxsjh-3","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.40-axxsjh-31","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1-axxsjh-49","obj_delete_at_hint.0000000047","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.2","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","datalog.sync-status.85494750-2c5c-47db-a993-372e454c1eb9","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-46:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24-axxsjh-15","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2-axxsjh-45","2020-03-28-07-2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1-axxsjh-42","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.1","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.2","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.8-axxsjh-57","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2-axxsjh-2","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29-axxsjh-20","obj_delete_at_hint.0000000052","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4-axxsjh-47","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-49:5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","obj_delete_at_hint.0000000041","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50-axxsjh-41","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49-axxsjh-40","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48-axxsjh-39","obj_delete_at_hint.0000000064","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33-axxsjh-24","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.37-axxsjh-28","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.3.retry","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.43-axxsjh-34","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","datalog.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28-axxsjh-19","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26-axxsjh-17","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-46:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21-axxsjh-12","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.0","mdlog.sync-status.shard.2","2020-03-28-07-2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2-axxsjh-43","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23-axxsjh-14","obj_delete_at_hint.0000000116","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45-axxsjh-36","obj_delete_at_hint.0000000034","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.1","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6-axxsjh-6","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.3","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30-axxsjh-21","obj_delete_at_hint.0000000073","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16-axxsjh-7","obj_delete_at_hint.0000000038","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32-axxsjh-23","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.0","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.2-axxsjh-51","mdlog.sync-status.shard.0","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","obj_delete_at_hint.0000000027","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35-axxsjh-26","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18-axxsjh-9","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1-axxsjh-44","obj_delete_at_hint.0000000039","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5-axxsjh-48","obj_delete_at_hint.0000000100","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.5-axxsjh-54","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5-axxsjh-5","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.1","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","obj_delete_at_hint.0000000012","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-48:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","obj_delete_at_hint.0000000020","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.1","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3-axxsjh-46","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22-axxsjh-13","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.3","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-48:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31-axxsjh-22","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.0","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34-axxsjh-25","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-47:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4","data.full-sync.index.85494750-2c5c-47db-a993-372e454c1eb9.3","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4-axxsjh-4","meta.full-sync.index.2","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-49:5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1","obj_delete_at_hint.0000000067","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.3","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.0","obj_delete_at_hint.0000000086","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20-axxsjh-11","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","datalog.sync-status.shard.85494750-2c5c-47db-a993-372e454c1eb9.3.retry","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","obj_delete_at_hint.0000000050","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.2","obj_delete_at_hint.0000000102","meta.log.0d07b3db-af62-4c83-b2b9-6aa7b82de92e.0","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","bucket.sync-status.85494750-2c5c-47db-a993-372e454c1eb9:axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","obj_delete_at_hint.0000000042","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.2","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19-axxsjh-10","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47-axxsjh-38","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1-axxsjh-1","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46-axxsjh-37"] 2020-03-28T07:35:47.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', u'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:47.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', u'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:47.192 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:35:48.736 INFO:teuthology.orchestra.run.smithi156.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.17-axxsjh-8","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","obj_delete_at_hint.0000000028","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.27-axxsjh-18","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25-axxsjh-16","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-47:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4","obj_delete_at_hint.0000000015","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.3-axxsjh-3","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1-axxsjh-49","obj_delete_at_hint.0000000047","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.3","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-46:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24-axxsjh-15","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.2-axxsjh-45","2020-03-28-07-2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.1-axxsjh-42","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.2-axxsjh-2","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29-axxsjh-20","obj_delete_at_hint.0000000052","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4-axxsjh-47","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","obj_delete_at_hint.0000000096","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.3.retry","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","obj_delete_at_hint.0000000041","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.50-axxsjh-41","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.49-axxsjh-40","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.38-axxsjh-29","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.48-axxsjh-39","obj_delete_at_hint.0000000064","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33-axxsjh-24","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-46:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.3.retry","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","datalog.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28-axxsjh-19","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.6-axxsjh-55","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.0","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26-axxsjh-17","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21-axxsjh-12","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.0","mdlog.sync-status.shard.2","2020-03-28-07-2c0a3e73-a280-4ccf-960d-d2c86331b6ff.8101.2-axxsjh-43","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23-axxsjh-14","obj_delete_at_hint.0000000116","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45-axxsjh-36","obj_delete_at_hint.0000000034","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-49:5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.6-axxsjh-6","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.3","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30-axxsjh-21","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-48:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5","obj_delete_at_hint.0000000073","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.16-axxsjh-7","obj_delete_at_hint.0000000038","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.2","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32-axxsjh-23","meta.full-sync.index.3","obj_delete_at_hint.0000000118","datalog.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.41-axxsjh-32","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","obj_delete_at_hint.0000000098","meta.log.3894ea80-31ba-4896-8e09-e661b246b356.0","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35-axxsjh-26","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.18-axxsjh-9","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.1-axxsjh-44","obj_delete_at_hint.0000000039","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5-axxsjh-48","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.3-axxsjh-52","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","obj_delete_at_hint.0000000100","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.5-axxsjh-5","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.1","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","obj_delete_at_hint.0000000012","meta.log.571a1648-7110-4ecb-afe2-66045679be41.0","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.2","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","obj_delete_at_hint.0000000020","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.1","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3-axxsjh-46","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22-axxsjh-13","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-48:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","obj_delete_at_hint.0000000013","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.1","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.44-axxsjh-35","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31-axxsjh-22","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-47:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34-axxsjh-25","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.4-axxsjh-4","meta.full-sync.index.2","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-49:5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1","obj_delete_at_hint.0000000067","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.3","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.0","obj_delete_at_hint.0000000086","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20-axxsjh-11","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.9-axxsjh-58","data.full-sync.index.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.0","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","obj_delete_at_hint.0000000050","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.2","obj_delete_at_hint.0000000102","meta.log.0d07b3db-af62-4c83-b2b9-6aa7b82de92e.0","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","obj_delete_at_hint.0000000124","meta.log.3894ea80-31ba-4896-8e09-e661b246b356.1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","datalog.sync-status.shard.2c0a3e73-a280-4ccf-960d-d2c86331b6ff.3","obj_delete_at_hint.0000000097","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","obj_delete_at_hint.0000000042","bucket.sync-status.2c0a3e73-a280-4ccf-960d-d2c86331b6ff:axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.2","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19-axxsjh-10","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47-axxsjh-38","2020-03-28-06-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.1-axxsjh-1","2020-03-28-07-5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46-axxsjh-37"] 2020-03-28T07:35:48.995 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['log', 'list', '--rgw-zone', u'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:48.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'log', 'list', '--rgw-zone', u'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:48.996 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:35:50.688 INFO:teuthology.orchestra.run.smithi168.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-47:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.4","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-17:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.26","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-46:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.3","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-15:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.24","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-11:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.20","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-36:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.45","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-24:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.33","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-20:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.29","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-14:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.23","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-25:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.34","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.3.retry","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","datalog.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.0","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.3","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","meta.log.3894ea80-31ba-4896-8e09-e661b246b356.0","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-22:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.31","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.1","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","obj_delete_at_hint.0000000012","meta.log.571a1648-7110-4ecb-afe2-66045679be41.0","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-12:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.21","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-21:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.30","obj_delete_at_hint.0000000020","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.1","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-38:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.47","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-10:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.19","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-48:5fb75e0a-cb77-479b-b462-3a1befb8717c.5072.5","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-19:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.28","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-26:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.35","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-16:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.25","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-37:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.46","meta.full-sync.index.2","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-49:5fb75e0a-cb77-479b-b462-3a1befb8717c.5257.1","obj_delete_at_hint.0000000067","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.3","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.0","obj_delete_at_hint.0000000086","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-23:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.32","obj_delete_at_hint.0000000050","data.full-sync.index.5fb75e0a-cb77-479b-b462-3a1befb8717c.2","obj_delete_at_hint.0000000102","meta.log.0d07b3db-af62-4c83-b2b9-6aa7b82de92e.0","obj_delete_at_hint.0000000124","meta.log.3894ea80-31ba-4896-8e09-e661b246b356.1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","bucket.sync-status.5fb75e0a-cb77-479b-b462-3a1befb8717c:axxsjh-13:5fb75e0a-cb77-479b-b462-3a1befb8717c.4391.22","obj_delete_at_hint.0000000042","datalog.sync-status.shard.5fb75e0a-cb77-479b-b462-3a1befb8717c.2","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-03-28T07:35:51.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:51.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:51.253 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:35:52.693 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380943.329177_178.1","last_update":"2020-03-28 07:35:43.329177Z"}] 2020-03-28T07:35:52.693 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-03-28T07:35:52.694 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:52.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:52.694 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:35:54.221 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:35:54.221 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:35:59.222 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:59.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:35:59.223 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:00.909 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:00.909 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:05.910 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:05.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:05.911 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:07.488 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:07.489 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:12.490 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:12.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:12.490 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:14.150 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:14.150 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:19.151 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:19.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:19.151 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:21.011 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:21.012 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:26.013 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:26.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:26.013 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:27.657 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:27.658 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:32.659 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:32.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:32.660 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:34.334 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:34.334 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:39.334 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:39.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:39.335 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:40.798 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:40.798 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:45.799 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:45.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:45.799 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:47.240 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:47.240 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:52.241 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:52.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:52.242 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:36:53.609 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:36:53.609 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:36:56.247 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:19:28.217 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:36:56.251 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:19:28.221 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:36:56.470 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:19:28.440 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:36:58.610 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:58.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:36:58.610 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:00.119 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:00.120 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:37:05.121 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:05.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:05.122 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:06.783 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:06.784 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:37:11.784 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:11.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:11.785 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:13.324 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:13.325 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:37:18.326 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:18.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:18.326 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:20.026 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:20.026 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:37:25.027 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:25.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:25.028 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:26.662 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:26.662 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:37:28.477 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:18:54.537 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:37:28.488 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:18:54.549 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:37:28.735 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:18:54.795 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:37:31.663 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:31.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:31.664 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:33.434 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380727.695793_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:32:07.695793Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:33.435 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=3 master=1_1585380943.329177_178.1 target=1_1585380727.695793_140.1 2020-03-28T07:37:38.435 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:38.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:38.436 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:40.273 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6895220985135347949},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380943.329177_178.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:35:43.329177Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:40.274 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-03-28T07:37:40.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:40.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:40.275 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:41.661 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380943.329177_178.1","last_update":"2020-03-28 07:35:43.329177Z"}] 2020-03-28T07:37:41.661 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-03-28T07:37:41.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:41.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:41.662 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:43.004 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":5750405467414497023},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380943.329177_178.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:35:43.329177Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:43.005 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-03-28T07:37:43.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:43.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:43.006 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:44.196 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380943.329177_178.1","last_update":"2020-03-28 07:35:43.329177Z"}] 2020-03-28T07:37:44.197 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-03-28T07:37:44.197 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:44.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:44.198 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:46.111 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3889226293267946400},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380943.329177_178.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:35:43.329177Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:46.112 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-03-28T07:37:46.112 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:46.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:46.112 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:47.450 INFO:teuthology.orchestra.run.smithi168.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380997.410291_179.1","last_update":"2020-03-28 07:36:37.410291Z"}] 2020-03-28T07:37:47.450 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-03-28T07:37:47.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:47.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:47.451 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:48.890 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":16017607255566841915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380997.410291_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:36:37.410291Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:48.891 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-03-28T07:37:48.891 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:48.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:48.891 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:50.385 INFO:teuthology.orchestra.run.smithi168.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380997.410291_179.1","last_update":"2020-03-28 07:36:37.410291Z"}] 2020-03-28T07:37:50.386 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-03-28T07:37:50.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:50.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:50.386 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:52.094 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2416084121408814850},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1585380997.410291_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:36:37.410291Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:52.094 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-03-28T07:37:52.094 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:52.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:52.095 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:53.470 INFO:teuthology.orchestra.run.smithi168.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1585380997.410291_179.1","last_update":"2020-03-28 07:36:37.410291Z"}] 2020-03-28T07:37:53.470 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone2 2020-03-28T07:37:53.471 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:53.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:37:53.471 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:37:55.365 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:37:55.365 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:00.366 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:00.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:00.367 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:02.460 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:02.461 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:07.462 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:07.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:07.462 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:09.514 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:09.514 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:14.515 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:14.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:14.516 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:16.529 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:16.529 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:21.530 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:21.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:21.531 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:23.392 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:23.393 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:28.394 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:28.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:28.395 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:30.548 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:30.549 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:35.550 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:35.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:35.550 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:37.526 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:37.526 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:42.527 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:42.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:42.528 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:44.749 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:44.749 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:49.749 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:49.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:49.750 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:51.733 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:51.733 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:38:56.734 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:56.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:38:56.735 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:38:58.487 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:38:58.487 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:03.488 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:03.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:03.489 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:05.722 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:05.723 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:10.724 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:10.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:10.725 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:12.761 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:12.761 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:17.762 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:17.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:17.763 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:19.708 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:19.709 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:24.710 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:24.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:24.711 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:27.180 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:27.180 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:32.182 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:32.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:32.182 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:34.123 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:34.123 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:39.124 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:39.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:39.125 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:41.080 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:41.080 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:46.081 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:46.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:46.082 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:48.209 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:48.210 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:39:53.210 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:53.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:39:53.211 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:39:55.214 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:39:55.214 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:00.215 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:00.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:00.216 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:02.445 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:02.445 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:07.446 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:07.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:07.447 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:09.595 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:09.596 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:14.596 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:14.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:14.597 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:16.614 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:16.614 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:21.615 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:21.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:21.616 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:23.585 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:23.585 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:28.586 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:28.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:28.586 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:30.776 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:30.777 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:35.777 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:35.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:35.778 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:38.029 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:38.030 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:43.031 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:43.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:43.032 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:45.292 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:45.293 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:50.293 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:50.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:50.294 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:52.503 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:52.503 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:40:57.504 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:57.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:40:57.505 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:40:59.389 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:40:59.390 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:04.391 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:04.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:04.392 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:05.978 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:05.978 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:10.978 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:10.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:10.979 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:12.532 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:12.533 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:17.534 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:17.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:17.535 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:19.170 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:19.170 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:24.171 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:24.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:24.171 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:25.737 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:25.737 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:30.738 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:30.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:30.738 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:32.341 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:32.342 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:33.225 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:24:05.194 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:41:33.228 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:24:05.198 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:41:33.417 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:24:05.386 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:41:37.342 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:37.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:37.343 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:38.982 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:38.982 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:43.983 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:43.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:43.984 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:46.076 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:46.076 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:51.077 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:51.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:51.077 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:41:53.379 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:41:53.379 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:41:58.380 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:58.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:41:58.381 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:00.534 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:00.534 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:05.535 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:05.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:05.536 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:07.607 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:07.607 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:12.607 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:12.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:12.608 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:14.628 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:14.629 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:19.630 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:19.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:19.631 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:21.704 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:21.704 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:26.705 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:26.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:26.705 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:28.829 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:28.829 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:33.831 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:33.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:33.831 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:35.758 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:35.758 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:39.623 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:24:05.684 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:42:39.667 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:24:05.728 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:42:39.848 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:24:05.908 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:42:40.759 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:40.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:40.759 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:43.424 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:43.424 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:48.425 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:48.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:48.426 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:50.635 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:50.635 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:42:55.636 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:55.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:42:55.637 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:42:57.548 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:42:57.549 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:02.549 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:02.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:02.550 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:04.538 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:04.538 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:09.539 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:09.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:09.540 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:11.625 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:11.626 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:16.626 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:16.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:16.627 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:18.561 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:18.561 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:23.562 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:23.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:23.563 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:25.658 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:25.659 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:30.660 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:30.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:30.660 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:32.711 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:32.711 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:37.712 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:37.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:37.713 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:39.965 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:39.965 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:44.966 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:44.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:44.967 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:47.184 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:47.184 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:50.982 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:26:22.953 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:43:51.003 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:26:22.973 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:43:51.165 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:26:23.135 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:43:52.185 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:52.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:52.185 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:43:54.233 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:43:54.233 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:43:59.234 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:59.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:43:59.235 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:01.481 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:01.482 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:06.483 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:06.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:06.484 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:08.703 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:08.704 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:13.705 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:13.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:13.706 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:15.847 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:15.847 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:20.849 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:20.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:20.850 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:22.895 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:22.895 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:23.966 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:25:50.027 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:44:23.974 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:25:50.034 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:44:24.176 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:25:50.236 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:44:27.895 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:27.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:27.897 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:30.013 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:30.013 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:35.014 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:35.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:35.015 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:37.293 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:37.294 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:42.294 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:42.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:42.295 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:44.274 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:44.274 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:49.275 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:49.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:44:49.275 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:44:51.220 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"init","num_shards":0,"instance_id":0},"markers":[]},"full_sync":{"total":0,"complete":0}} 2020-03-28T07:44:51.221 ERROR:rgw_multi.tests:len(log_status)=4 len(sync_status)=0 2020-03-28T07:44:56.222 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... FAIL 2020-03-28T07:44:56.223 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-59 2020-03-28T07:44:56.833 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-60 2020-03-28T07:44:58.313 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-61 2020-03-28T07:45:04.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:04.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:04.225 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:07.660 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.20.5"}]} 2020-03-28T07:45:07.661 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-59 2020-03-28T07:45:07.661 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:07.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:07.661 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:09.199 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.20.5"}}}] 2020-03-28T07:45:09.200 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-59 2020-03-28T07:45:09.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:09.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:09.200 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:10.633 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.20.5"}]} 2020-03-28T07:45:10.634 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-59 2020-03-28T07:45:10.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:10.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:10.634 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:12.052 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.20.5"}}}] 2020-03-28T07:45:12.053 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-59 2020-03-28T07:45:12.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:12.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:12.053 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:13.475 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.20.5"}]} 2020-03-28T07:45:13.476 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-59 2020-03-28T07:45:13.476 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:13.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:13.477 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:15.777 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.20.5"}}}] 2020-03-28T07:45:15.777 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-59 2020-03-28T07:45:15.778 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:15.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:15.779 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:17.261 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.18.5"}]} 2020-03-28T07:45:17.261 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-59 2020-03-28T07:45:17.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:17.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:17.262 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:19.306 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.18.5"}}}] 2020-03-28T07:45:19.307 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-59 2020-03-28T07:45:19.307 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:19.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:19.307 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:20.791 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.18.5"}]} 2020-03-28T07:45:20.791 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-59 2020-03-28T07:45:20.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:20.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:20.792 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:22.593 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.18.5"}}}] 2020-03-28T07:45:22.594 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-59 2020-03-28T07:45:22.594 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:22.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:22.594 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:24.092 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.18.5"}]} 2020-03-28T07:45:24.092 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-59 2020-03-28T07:45:24.093 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:24.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:24.093 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:26.094 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:45:26.094 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:45:31.095 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:31.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:31.095 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:33.600 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:45:33.601 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:45:38.601 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:38.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:38.602 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:40.859 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:45:40.860 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:45:45.860 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:45.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:45.861 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:48.135 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:45:48.135 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:45:53.136 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:53.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:45:53.137 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:45:55.438 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:45:55.438 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:00.439 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:00.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:00.440 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:02.493 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:02.494 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:07.494 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:07.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:07.495 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:09.693 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:09.693 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:14.694 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:14.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:14.695 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:16.911 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:16.911 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:21.912 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:21.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:21.913 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:24.124 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:24.124 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:29.125 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:29.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:29.126 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:31.654 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:31.655 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:36.655 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:36.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:36.656 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:38.895 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:38.895 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:43.896 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:43.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:43.897 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:46.011 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:46.011 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:51.012 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:51.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:51.013 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:46:53.074 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:46:53.074 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:46:58.075 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:58.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:46:58.075 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:00.445 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:00.446 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:05.447 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:05.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:05.448 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:07.804 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:07.804 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:12.805 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:12.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:12.805 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:15.010 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:15.036 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:20.036 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:20.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:20.037 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:22.323 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:22.323 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:27.325 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:27.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:27.325 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:29.443 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:29.443 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:34.444 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:34.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:34.445 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:36.452 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:36.453 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:41.454 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:41.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:41.454 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:43.690 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:43.691 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:48.692 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:48.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:48.692 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:51.209 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:51.209 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:47:56.211 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:56.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:47:56.211 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:47:58.494 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:47:58.494 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:03.495 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:03.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:03.495 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:05.742 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:05.742 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:10.743 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:10.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:10.744 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:13.232 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:13.233 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:18.234 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:18.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:18.235 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:20.334 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:20.334 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:25.335 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:25.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:25.336 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:27.470 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:27.471 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:32.471 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:32.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:32.472 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:34.716 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:34.716 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:39.717 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:39.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:39.718 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:42.115 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:42.115 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:47.116 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:47.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:47.117 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:49.471 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:49.471 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:48:54.472 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:54.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:48:54.473 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:48:57.096 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:48:57.096 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:02.097 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:02.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:02.097 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:04.260 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:04.260 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:09.261 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:09.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:09.262 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:11.469 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:11.469 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:16.470 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:16.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:16.471 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:18.932 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:18.932 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:23.933 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:23.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:23.934 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:26.094 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:26.095 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:31.095 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:31.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:31.096 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:33.560 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:33.560 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:38.562 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:38.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:38.562 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:40.889 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:40.889 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:45.890 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:45.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:45.891 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:48.109 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:48.109 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:49:53.109 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:53.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:49:53.111 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:49:54.137 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:32:26.107 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:49:54.142 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:32:26.112 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:49:54.307 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:32:26.278 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:49:55.332 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:49:55.333 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:00.333 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:00.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:00.334 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:03.145 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:03.146 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:08.147 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:08.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:08.147 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:10.663 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:10.664 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:15.665 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:15.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:15.665 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:17.714 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:17.715 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:22.715 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:22.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:22.716 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:24.800 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:24.800 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:29.803 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:29.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:29.803 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:32.233 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:32.234 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:37.235 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:37.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:37.235 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:39.513 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:39.514 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:44.515 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:44.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:44.515 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:46.645 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:46.645 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:47.525 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:33:19.494 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:50:47.549 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:33:19.519 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:50:47.625 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:33:19.595 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:50:51.646 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:51.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:51.646 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:50:54.345 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:50:54.346 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:50:59.346 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:59.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:50:59.347 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:00.654 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:32:26.714 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:51:00.662 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:32:26.722 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:51:00.855 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:32:26.915 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:51:01.643 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:01.644 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:06.644 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:06.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:06.645 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:08.935 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:08.935 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:13.936 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:13.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:13.936 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:16.440 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:16.441 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:19.356 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:32:45.416 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:51:19.376 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:32:45.436 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:51:19.516 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:32:45.576 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:51:21.441 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:21.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:21.442 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:23.562 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:23.563 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:28.564 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:28.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:28.564 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:30.884 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:30.884 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:35.885 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:35.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:35.886 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:38.123 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:38.124 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:43.124 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:43.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:43.125 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:45.436 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:45.436 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:50.437 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:50.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:50.438 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:52.654 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:52.655 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:51:57.656 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:57.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:51:57.657 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:51:59.750 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:51:59.750 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:52:04.751 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:04.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:04.752 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:07.104 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:52:07.105 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:52:12.106 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:12.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:12.106 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:14.179 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:52:14.180 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:52:19.181 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:19.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:19.182 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:21.422 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:52:21.422 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:52:26.423 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:26.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:26.424 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:28.653 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:52:28.653 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:52:33.654 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:33.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:33.655 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:36.263 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:52:36.263 WARNING:rgw_multi.tests:bucket axxsjh-59 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.18.5 target= 2020-03-28T07:52:41.265 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2020-03-28T07:52:41.265 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-62 2020-03-28T07:52:41.895 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-63 2020-03-28T07:52:43.047 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-64 2020-03-28T07:52:47.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:47.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:47.815 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:49.534 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T07:52:50.877 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585381963.855404_181.1","last_update":"2020-03-28 07:52:43.855404Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T07:52:50.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T07:52:50.878 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:50.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:50.879 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:52.780 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585381963.855404_181.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:52:43.855404Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:52:52.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T07:52:52.781 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T07:52:52.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:52.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:52.781 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:54.214 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585381963.855404_181.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:52:43.855404Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:52:54.215 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T07:52:54.215 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T07:52:54.215 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:54.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:54.216 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:56.541 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585381963.855404_181.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 07:52:43.855404Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T07:52:56.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T07:52:56.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:56.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:56.542 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:58.073 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T07:52:58.074 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-62 2020-03-28T07:52:58.074 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:58.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:58.074 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:52:59.562 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.8.5"}}}] 2020-03-28T07:52:59.562 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-62 2020-03-28T07:52:59.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:59.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:52:59.563 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:00.968 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T07:53:00.968 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-62 2020-03-28T07:53:00.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:00.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:00.969 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:02.354 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.8.5"}}}] 2020-03-28T07:53:02.355 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-62 2020-03-28T07:53:02.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:02.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:02.356 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:03.605 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T07:53:03.606 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-62 2020-03-28T07:53:03.606 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:03.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:03.606 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:06.241 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.8.5"}}}] 2020-03-28T07:53:06.242 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-62 2020-03-28T07:53:06.242 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:06.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:06.242 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:07.889 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T07:53:07.889 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-62 2020-03-28T07:53:07.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:07.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:07.890 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:09.587 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.8.5"}}}] 2020-03-28T07:53:09.587 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-62 2020-03-28T07:53:09.587 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:09.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:09.588 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:11.089 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T07:53:11.090 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-62 2020-03-28T07:53:11.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:11.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:11.090 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:12.756 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.8.5"}}}] 2020-03-28T07:53:12.756 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-62 2020-03-28T07:53:12.756 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:12.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:12.757 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:14.188 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.8.5"}]} 2020-03-28T07:53:14.188 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-62 2020-03-28T07:53:14.189 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:14.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:14.189 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:16.482 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:53:16.483 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:53:21.484 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:21.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:21.485 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:24.025 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:53:24.025 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:53:29.026 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:29.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:29.027 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:31.608 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:53:31.609 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:53:36.610 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:36.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:36.611 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:39.161 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:53:39.161 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:53:44.163 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:44.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:44.163 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:46.273 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:53:46.274 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:53:51.275 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:51.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:51.275 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:53:53.661 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:53:53.662 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:53:58.663 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:58.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:53:58.664 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:01.057 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:01.058 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:06.059 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:06.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:06.059 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:08.528 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:08.529 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:13.530 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:13.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:13.530 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:16.131 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:16.132 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:21.133 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:21.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:21.133 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:23.695 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:23.696 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:28.697 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:28.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:28.697 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:30.932 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:30.933 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:35.934 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:35.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:35.934 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:38.411 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:38.411 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:43.411 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:43.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:43.412 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:45.829 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:45.829 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:50.830 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:50.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:50.831 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:54:53.242 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:54:53.242 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:54:58.243 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:58.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:54:58.244 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:00.334 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:00.334 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:05.335 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:05.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:05.335 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:07.886 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:07.887 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:12.887 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:12.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:12.888 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:15.182 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:15.182 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:20.184 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:20.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:20.184 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:22.461 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:22.462 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:27.463 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:27.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:27.464 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:29.791 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:29.791 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:34.792 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:34.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:34.792 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:37.210 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:37.210 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:42.211 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:42.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:42.212 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:44.490 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:44.491 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:49.492 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:49.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:49.492 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:51.869 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:51.869 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:55:56.870 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:56.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:55:56.870 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:55:59.144 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:55:59.145 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:04.146 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:04.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:04.146 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:06.811 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:06.811 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:11.812 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:11.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:11.812 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:13.894 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:13.894 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:18.896 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:18.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:18.896 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:21.534 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:21.534 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:26.535 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:26.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:26.536 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:28.934 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:28.934 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:33.935 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:33.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:33.936 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:36.413 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:36.413 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:41.414 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:41.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:41.415 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:43.708 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:43.708 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:48.709 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:48.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:48.710 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:51.097 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:51.097 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:56:56.099 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:56.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:56:56.099 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:56:58.589 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:56:58.589 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:03.590 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:03.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:03.591 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:05.552 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:05.553 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:10.554 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:10.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:10.555 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:13.219 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:13.219 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:18.221 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:18.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:18.221 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:20.708 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:20.708 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:25.710 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:25.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:25.711 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:28.072 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:28.072 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:33.073 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:33.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:33.074 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:35.569 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:35.569 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:40.570 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:40.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:40.571 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:42.911 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:42.912 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:44.087 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:40:16.058 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:57:44.094 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:40:16.064 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:57:44.544 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:40:16.514 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:57:47.912 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:47.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:47.913 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:50.545 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:50.546 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:57:55.547 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:55.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:57:55.548 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:57:57.840 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:57:57.840 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:02.841 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:02.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:02.841 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:05.089 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:05.089 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:10.090 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:10.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:10.091 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:12.321 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:12.322 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:15.151 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:40:47.122 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:58:15.157 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:40:47.128 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:58:15.210 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:40:47.180 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T07:58:15.794 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:39:41.853 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:58:15.798 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:39:41.858 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:58:16.127 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:39:42.187 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:58:17.323 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:17.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:17.323 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:20.275 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:20.275 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:25.276 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:25.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:25.276 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:27.484 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:27.484 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:32.485 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:32.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:32.485 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:34.834 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:34.835 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:39.836 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:39.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:39.837 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:41.972 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:41.972 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:46.973 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:46.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:46.974 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:49.339 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:49.339 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:58:54.339 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:54.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:58:54.340 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:58:56.697 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:58:56.698 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:01.698 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:01.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:01.699 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:04.254 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:04.255 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:09.256 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:09.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:09.256 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:11.643 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:11.643 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:16.644 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:16.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:16.645 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:19.089 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:19.090 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:22.405 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:40:48.465 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:59:22.410 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:40:48.470 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:59:22.484 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:40:48.543 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T07:59:24.091 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:24.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:24.091 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:26.881 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:26.882 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:31.883 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:31.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:31.884 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:34.554 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:34.555 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:39.556 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:39.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:39.556 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:41.799 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:41.800 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:46.801 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:46.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:46.801 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:49.117 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:49.118 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T07:59:54.118 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:54.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T07:59:54.119 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T07:59:56.620 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T07:59:56.620 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T08:00:01.622 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:01.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:01.622 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:00:04.190 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:00:04.190 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T08:00:09.191 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:09.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:09.192 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:00:11.584 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:00:11.585 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T08:00:16.587 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:16.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:16.588 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:00:18.883 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:00:18.883 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T08:00:23.884 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:23.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:23.885 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:00:25.976 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:00:25.977 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T08:00:30.977 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:30.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:30.978 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:00:33.225 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:00:33.226 WARNING:rgw_multi.tests:bucket axxsjh-62 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.8.5 target= 2020-03-28T08:00:38.227 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2020-03-28T08:00:38.227 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-65 2020-03-28T08:00:38.900 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=axxsjh-66 2020-03-28T08:00:40.011 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=axxsjh-67 2020-03-28T08:00:57.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:57.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:57.046 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:00:58.549 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000018.32.5"}]} 2020-03-28T08:00:58.549 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-65 2020-03-28T08:00:58.550 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:58.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:00:58.550 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:00.225 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000018.32.5"}}}] 2020-03-28T08:01:00.225 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-65 2020-03-28T08:01:00.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:00.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:00.226 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:01.581 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000018.32.5"}]} 2020-03-28T08:01:01.581 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-65 2020-03-28T08:01:01.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:01.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:01.582 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:02.800 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000018.32.5"}}}] 2020-03-28T08:01:02.800 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-65 2020-03-28T08:01:02.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:02.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:02.801 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:04.139 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000018.32.5"}]} 2020-03-28T08:01:04.140 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-65 2020-03-28T08:01:04.140 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:04.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:04.140 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:06.828 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000018.32.5"}}}] 2020-03-28T08:01:06.828 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-65 2020-03-28T08:01:06.828 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:06.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:06.829 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:08.265 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.16.5"}]} 2020-03-28T08:01:08.265 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-65 2020-03-28T08:01:08.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:08.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:08.266 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:09.975 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.16.5"}}}] 2020-03-28T08:01:09.975 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-65 2020-03-28T08:01:09.975 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:09.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:09.976 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:11.377 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.16.5"}]} 2020-03-28T08:01:11.378 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-65 2020-03-28T08:01:11.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:11.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:11.379 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:12.974 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"0#00000000002.16.5"}}}] 2020-03-28T08:01:12.975 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-65 2020-03-28T08:01:12.975 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:12.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:12.976 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:14.558 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000002.16.5"}]} 2020-03-28T08:01:14.558 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-65 2020-03-28T08:01:14.559 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:14.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:14.559 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:16.892 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:01:16.893 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:01:21.893 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:21.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:21.894 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:24.112 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:01:24.112 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:01:29.113 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:29.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:29.113 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:31.664 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:01:31.664 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:01:36.665 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:36.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:36.665 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:38.832 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:01:38.832 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:01:43.833 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:43.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:43.834 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:46.255 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:01:46.255 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:01:51.256 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:51.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:51.257 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:01:53.670 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:01:53.670 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:01:58.671 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:58.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:01:58.671 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:00.680 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:00.681 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:05.681 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:05.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:05.682 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:08.140 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:08.140 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:13.141 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:13.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:13.142 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:15.178 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:15.178 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:20.180 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:20.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:20.180 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:22.281 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:22.282 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:27.283 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:27.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:27.283 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:29.734 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:29.734 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:34.735 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:34.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:34.736 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:36.813 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:36.814 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:41.815 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:41.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:41.816 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:44.122 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:44.123 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:49.123 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:49.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:49.124 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:51.669 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:51.669 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:02:56.670 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:56.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:02:56.671 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:02:59.015 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:02:59.016 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:04.017 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:04.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:04.017 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:06.185 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:06.186 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:11.187 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:11.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:11.187 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:13.495 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:13.496 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:18.497 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:18.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:18.497 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:21.093 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:21.093 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:26.095 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:26.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:26.095 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:28.569 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:28.570 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:33.571 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:33.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:33.571 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:36.091 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:36.091 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:41.092 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:41.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:41.093 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:43.294 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:43.294 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:48.296 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:48.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:48.296 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:50.611 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:50.611 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:03:55.612 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:55.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:03:55.612 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:03:58.087 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:03:58.087 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:03.088 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:03.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:03.088 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:05.523 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:05.523 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:10.525 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:10.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:10.526 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:12.809 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:12.809 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:17.810 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:17.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:17.811 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:20.080 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:20.081 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:20.518 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:46:52.489 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:04:20.525 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:46:52.496 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:04:20.861 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:46:52.832 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:04:25.081 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:25.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:25.082 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:27.551 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:27.551 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:32.552 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:32.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:32.552 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:34.825 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:34.825 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:39.826 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:39.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:39.827 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:42.382 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:42.383 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:47.384 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:47.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:47.385 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:49.925 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:49.925 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:04:54.926 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:54.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:04:54.927 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:04:57.605 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:04:57.606 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:02.607 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:02.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:02.608 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:04.961 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:04.961 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:09.962 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:09.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:09.962 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:12.120 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:46:38.179 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:05:12.127 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:46:38.187 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:05:12.203 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:12.204 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:12.517 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:46:38.577 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:05:17.205 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:17.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:17.205 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:19.536 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:19.536 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:24.538 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:24.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:24.538 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:26.891 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:26.891 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:31.892 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:31.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:31.893 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:34.282 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:34.282 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:39.283 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:39.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:39.284 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:41.714 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:41.714 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:46.715 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:46.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:46.716 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:49.340 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:49.340 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:05:54.341 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:54.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:05:54.342 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:05:56.901 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:05:56.902 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:01.903 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:01.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:01.904 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:04.317 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:04.318 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:09.319 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:09.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:09.319 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:11.528 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:11.529 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:16.530 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:16.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:16.530 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:19.031 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:19.031 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:24.032 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:24.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:24.032 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:26.130 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:26.130 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:31.131 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:31.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:31.132 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:33.527 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:33.528 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:36.729 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:49:08.699 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:36.732 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:49:08.703 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:36.872 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:49:08.843 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:38.529 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:38.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:38.530 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:40.912 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:40.913 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:45.914 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:45.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:45.914 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:48.247 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:48.248 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:06:51.978 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:49:23.948 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:52.058 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:49:24.029 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:52.112 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:49:24.083 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:53.248 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:53.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:06:53.249 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:06:53.572 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:48:19.632 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:53.630 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:48:19.690 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:53.765 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:48:19.825 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:06:56.165 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:06:56.165 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:01.167 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:01.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:01.167 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:03.522 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:03.522 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:08.523 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:08.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:08.525 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:11.135 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:11.135 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:16.136 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:16.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:16.136 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:18.382 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:18.382 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:23.383 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:23.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:23.384 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:25.725 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:25.725 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:30.726 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:30.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:30.726 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:33.406 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:33.406 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:38.406 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:38.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:38.407 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:41.159 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:41.159 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:43.862 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:49:09.921 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:07:43.867 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:49:09.927 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:07:43.951 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:49:10.010 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:07:46.160 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:46.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:46.161 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:48.435 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:48.435 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:07:53.436 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:53.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:07:53.437 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:07:55.907 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:07:55.908 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:08:00.909 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:00.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:00.909 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:03.417 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:08:03.418 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:08:08.418 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:08.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:08.419 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:10.577 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:08:10.577 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:08:15.578 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:15.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:15.579 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:18.111 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:08:18.111 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:08:23.113 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:23.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:23.113 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:25.611 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:08:25.612 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:08:30.612 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:30.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:30.613 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:33.400 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:08:33.400 WARNING:rgw_multi.tests:bucket axxsjh-65 zone test-zone4 behind zone test-zone2: shard=0 master=00000000002.16.5 target= 2020-03-28T08:08:38.402 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... FAIL 2020-03-28T08:08:38.402 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-68 2020-03-28T08:08:39.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:39.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:39.874 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:41.094 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:08:41.511 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585382918.865385_205.1","last_update":"2020-03-28 08:08:38.865385Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:08:41.511 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:08:41.511 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:41.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:41.512 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:42.884 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382918.865385_205.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:38.865385Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:08:42.884 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:08:42.884 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:08:42.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:42.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:42.885 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:44.421 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382918.865385_205.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:38.865385Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:08:44.422 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:08:44.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:08:44.422 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:44.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:44.422 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:46.742 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382918.865385_205.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:38.865385Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:08:46.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:08:46.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:46.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:46.743 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-68 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:48.080 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000004.4.5"}]} 2020-03-28T08:08:48.080 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-68 2020-03-28T08:08:48.080 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:48.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:48.081 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-68 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:49.833 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":2},"inc_marker":{"position":"0#00000000004.4.5"}}}] 2020-03-28T08:08:49.834 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-68 2020-03-28T08:08:50.569 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-03-28T08:08:50.569 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=axxsjh-69 2020-03-28T08:08:52.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:52.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:52.115 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:53.273 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:08:53.986 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585382931.087686_209.1","last_update":"2020-03-28 08:08:51.087686Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:08:53.987 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:08:53.987 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:53.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:53.988 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:55.511 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382931.087686_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:51.087686Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:08:55.511 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:08:55.511 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:08:55.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:55.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:55.512 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:56.983 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382931.087686_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:51.087686Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:08:56.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:08:56.984 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:08:56.984 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:56.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:56.985 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:08:59.326 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382931.087686_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:51.087686Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:08:59.327 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:08:59.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:59.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:08:59.327 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:00.787 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.13.5"}]} 2020-03-28T08:09:00.788 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-69 2020-03-28T08:09:00.788 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:00.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:00.789 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:02.527 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.13.5"}}}] 2020-03-28T08:09:02.528 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-69 2020-03-28T08:09:02.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:02.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:02.528 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:03.986 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.13.5"}]} 2020-03-28T08:09:03.987 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-69 2020-03-28T08:09:03.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:03.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:03.987 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:05.289 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.13.5"}}}] 2020-03-28T08:09:05.289 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-69 2020-03-28T08:09:05.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:05.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:05.290 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:06.675 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000008.13.5"}]} 2020-03-28T08:09:06.675 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-69 2020-03-28T08:09:06.675 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:06.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:06.676 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:09.108 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.13.5"}}}] 2020-03-28T08:09:09.108 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-69 2020-03-28T08:09:09.108 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:09.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:09.109 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:10.666 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.12.5"}]} 2020-03-28T08:09:10.667 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-69 2020-03-28T08:09:10.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:10.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:10.667 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:12.348 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.12.5"}}}] 2020-03-28T08:09:12.348 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-69 2020-03-28T08:09:12.349 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:12.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:12.349 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:13.860 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.12.5"}]} 2020-03-28T08:09:13.861 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-69 2020-03-28T08:09:13.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:13.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:13.861 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:15.279 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":4},"inc_marker":{"position":"0#00000000008.12.5"}}}] 2020-03-28T08:09:15.279 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-69 2020-03-28T08:09:15.279 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:15.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:15.280 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:16.723 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000008.12.5"}]} 2020-03-28T08:09:16.724 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-69 2020-03-28T08:09:16.724 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:16.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:16.725 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:19.272 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:09:19.273 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:09:24.273 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:24.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:24.274 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:26.918 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:09:26.919 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:09:31.920 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:31.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:31.920 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:34.349 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:09:34.350 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:09:39.351 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:39.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:39.352 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:42.053 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:09:42.053 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:09:47.054 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:47.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:47.055 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:49.520 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:09:49.520 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:09:54.521 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:54.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:09:54.521 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:09:56.985 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:09:56.985 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:01.986 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:01.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:01.987 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:04.482 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:04.482 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:09.483 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:09.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:09.484 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:11.869 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:11.870 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:16.871 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:16.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:16.871 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:19.262 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:19.263 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:24.264 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:24.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:24.264 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:26.564 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:26.564 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:31.565 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:31.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:31.566 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:34.022 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:34.022 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:39.023 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:39.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:39.024 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:41.357 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:41.357 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:46.358 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:46.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:46.371 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:49.063 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:49.063 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:10:54.064 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:54.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:10:54.065 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:10:56.500 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:10:56.500 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:01.501 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:01.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:01.501 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:04.095 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:04.096 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:09.097 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:09.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:09.098 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:11.591 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:11.591 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:14.844 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:53:46.815 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:11:14.850 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:53:46.821 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:11:15.160 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:53:47.131 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:11:16.592 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:16.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:16.593 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:19.108 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:19.108 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:24.109 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:24.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:24.110 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:26.744 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:26.744 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:31.745 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:31.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:31.746 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:34.435 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:34.436 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:39.436 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:39.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:39.437 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:41.880 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:41.880 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:46.881 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:46.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:46.882 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:49.407 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:49.407 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:11:54.408 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:54.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:11:54.409 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:11:56.690 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:11:56.691 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:01.692 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:01.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:01.693 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:04.257 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:04.257 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:04.613 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:53:30.673 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:12:04.627 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:53:30.687 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:12:04.940 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:53:30.999 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:12:09.258 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:09.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:09.259 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:11.785 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:11.786 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:16.787 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:16.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:16.787 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:19.116 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:19.116 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:24.117 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:24.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:24.118 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:26.706 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:26.706 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:31.707 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:31.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:31.707 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:34.235 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:34.236 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:39.237 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:39.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:39.238 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:41.873 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:41.873 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:46.873 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:46.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:46.876 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:49.557 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:49.558 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:12:54.559 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:54.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:12:54.559 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:12:56.918 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:12:56.919 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:01.920 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:01.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:01.921 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:04.360 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:04.361 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:09.362 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:09.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:09.362 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:12.079 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:12.079 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:17.080 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:17.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:17.081 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:19.690 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:19.691 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:24.692 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:24.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:24.693 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:26.964 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:26.965 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:31.966 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:31.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:31.966 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:34.323 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:34.324 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:39.325 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:39.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:39.326 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:41.816 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:41.816 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:46.817 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:46.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:46.817 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:49.352 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:49.353 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:13:54.353 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:54.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:13:54.354 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:13:56.981 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:13:56.981 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:01.982 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:01.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:01.982 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:04.450 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:04.450 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:09.451 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:09.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:09.452 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:11.903 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:11.904 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:16.905 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:16.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:16.906 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:19.389 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:19.389 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:24.391 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:24.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:24.391 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:26.925 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:26.925 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:31.926 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:31.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:31.926 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:34.487 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:34.487 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:39.488 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:39.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:39.489 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:42.070 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:42.071 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:47.072 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:47.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:47.073 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:50.027 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:50.028 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:55.029 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:55.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:14:55.030 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:14:57.623 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:14:57.624 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:14:57.843 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:57:29.814 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:14:57.846 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:57:29.817 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:14:57.968 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:01:57:29.939 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:15:02.625 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:02.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:02.625 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:05.117 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:05.118 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:15:10.118 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:10.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:10.119 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:12.745 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:12.745 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:15:17.747 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:17.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:17.747 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:20.268 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:20.269 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:15:25.270 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:25.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:25.271 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:27.629 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:27.629 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:15:32.631 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:32.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:32.631 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:35.519 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:35.520 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:15:40.521 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:40.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:40.521 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:43.060 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:43.061 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:15:48.062 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:48.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:48.063 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:50.437 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:50.438 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:15:55.439 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:55.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:15:55.439 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:15:57.810 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:15:57.810 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:16:02.811 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:02.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:02.811 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:04.732 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:57:30.791 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:16:04.746 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:57:30.806 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:16:04.937 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:01:57:30.997 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:16:05.649 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:16:05.649 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:16:10.650 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:10.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:10.651 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:13.173 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:16:13.173 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:16:18.175 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:18.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:18.175 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:20.640 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:16:20.640 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:16:25.641 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:25.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:25.642 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:28.511 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:16:28.511 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:16:33.511 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:33.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:33.512 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:36.431 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:16:36.432 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:16:41.432 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:41.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:41.433 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:43.922 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:16:43.922 WARNING:rgw_multi.tests:bucket axxsjh-69 zone test-zone4 behind zone test-zone2: shard=0 master=00000000008.12.5 target= 2020-03-28T08:16:48.927 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2020-03-28T08:16:49.654 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-03-28T08:16:49.654 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.654 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.655 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_object_sync 2020-03-28T08:16:49.655 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.655 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.655 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.655 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.655 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 623, in test_object_sync 2020-03-28T08:16:49.655 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-11 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_object_delete 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.656 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 647, in test_object_delete 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-14 2020-03-28T08:16:49.657 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_versioned_object_incremental_sync 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 685, in test_versioned_object_incremental_sync 2020-03-28T08:16:49.658 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-17 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.659 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 737, in test_concurrent_versioned_object_incremental_sync 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.660 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-19 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_version_suspended_incremental_sync 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.661 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 765, in test_version_suspended_incremental_sync 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-20 2020-03-28T08:16:49.662 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_delete_marker_full_sync 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 808, in test_delete_marker_full_sync 2020-03-28T08:16:49.663 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-21 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.664 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 832, in test_suspended_delete_marker_full_sync 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.665 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-24 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_datalog_autotrim 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.666 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 979, in test_datalog_autotrim 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 384, in zonegroup_data_checkpoint 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 376, in zone_data_checkpoint 2020-03-28T08:16:49.667 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name) 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone4 source_zone=test-zone2 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.668 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 1127, in test_bucket_sync_disable 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 384, in zonegroup_data_checkpoint 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 376, in zone_data_checkpoint 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name) 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone4 source_zone=test-zone2 2020-03-28T08:16:49.669 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 1143, in test_bucket_sync_enable_right_after_disable 2020-03-28T08:16:49.670 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-59 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.671 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable_enable 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 1177, in test_bucket_sync_disable_enable 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.672 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-62 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_multipart_object_sync 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.673 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 1218, in test_multipart_object_sync 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.674 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-65 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_index_log_trim 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:16:49.675 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 1284, in test_bucket_index_log_trim 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 1280, in make_test_bucket 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 414, in zonegroup_bucket_checkpoint 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 406, in zone_bucket_checkpoint 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name, bucket_name) 2020-03-28T08:16:49.676 INFO:tasks.rgw_multisite_tests:AssertionError: failed bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-69 2020-03-28T08:16:49.677 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.677 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:16:49.677 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 6675.056s 2020-03-28T08:16:49.677 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:16:49.677 INFO:tasks.rgw_multisite_tests:FAILED (failures=13) 2020-03-28T08:16:49.682 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-03-28T08:16:49.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:49.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:49.683 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:50.703 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:16:51.048 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585382931.087686_209.1","last_update":"2020-03-28 08:08:51.087686Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:16:51.049 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:16:51.049 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:51.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:51.050 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:52.541 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382931.087686_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:51.087686Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:16:52.541 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:16:52.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:16:52.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:52.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:52.542 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:54.123 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382931.087686_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:51.087686Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:16:54.123 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:16:54.124 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:16:54.124 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:54.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:54.124 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:56.592 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382931.087686_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:51.087686Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:16:56.593 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:16:56.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:56.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:56.594 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:16:57.638 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:16:57.992 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585382931.087686_209.1","last_update":"2020-03-28 08:08:51.087686Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:16:57.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:16:57.992 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:57.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:16:57.993 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:00.379 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585382931.087686_209.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:08:51.087686Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:00.379 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:00.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:00.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:00.939 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:02.131 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:02.421 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383420.846401_217.1","last_update":"2020-03-28 08:17:00.846401Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:02.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:02.422 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:02.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:02.422 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:04.973 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383420.846401_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:00.846401Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:04.974 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:06.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:06.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:06.401 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:07.449 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:07.769 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383420.846401_217.1","last_update":"2020-03-28 08:17:00.846401Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:07.769 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:07.769 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:07.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:07.770 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:10.357 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383420.846401_217.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:00.846401Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:10.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:11.570 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-03-28T08:17:11.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:11.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:11.570 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:12.789 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:13.139 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383431.447603_219.1","last_update":"2020-03-28 08:17:11.447603Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:13.139 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:17:13.139 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:13.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:13.140 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:14.660 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383431.447603_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:11.447603Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:14.660 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:17:14.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:17:14.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:14.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:14.661 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:16.001 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383431.447603_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:11.447603Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:16.001 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:17:16.001 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:16.002 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:16.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:16.002 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:18.558 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383431.447603_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:11.447603Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:18.558 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:18.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:18.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:18.559 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:19.772 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:20.137 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383431.447603_219.1","last_update":"2020-03-28 08:17:11.447603Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:20.138 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:20.138 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:20.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:20.139 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:22.829 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383431.447603_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:11.447603Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:22.830 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:23.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:23.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:23.321 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:24.275 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:24.629 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383443.252597_223.1","last_update":"2020-03-28 08:17:23.252597Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:24.630 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:24.630 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:24.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:24.630 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:27.363 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383443.252597_223.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:23.252597Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:27.363 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:27.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:27.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:27.796 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:28.836 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:29.128 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383443.252597_223.1","last_update":"2020-03-28 08:17:23.252597Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:29.128 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:29.129 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:29.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:29.129 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:31.699 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383443.252597_223.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:23.252597Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:31.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:34.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:34.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:34.291 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-71 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:37.036 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000023.23.3"}]} 2020-03-28T08:17:37.036 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-71 2020-03-28T08:17:37.036 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-71', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:37.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-71', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:37.037 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-71 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:39.975 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"8","instance":"","ns":""},"count":9},"inc_marker":{"position":"00000000023.23.3"}}}] 2020-03-28T08:17:39.975 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-71 2020-03-28T08:17:40.155 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383453.865422.ce7fc0c6', u'eventTime': u'2020-03-28 08:17:32.568387Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'4', u'versionId': u'', u'etag': u'ce7fc0c6', u'sequencer': u'1D087F5E494B9533', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.155 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383453.977745.cfa76550', u'eventTime': u'2020-03-28 08:17:32.420186Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'3', u'versionId': u'', u'etag': u'cfa76550', u'sequencer': u'1D087F5E8A35473A', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.155 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383454.095524.2df8d975', u'eventTime': u'2020-03-28 08:17:31.868715Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'0', u'versionId': u'', u'etag': u'2df8d975', u'sequencer': u'1E087F5E7A96B105', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.155 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383454.153384.d973973d', u'eventTime': u'2020-03-28 08:17:32.895488Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'6', u'versionId': u'', u'etag': u'd973973d', u'sequencer': u'1E087F5E13752409', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.155 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383454.281925.8f2cfdea', u'eventTime': u'2020-03-28 08:17:32.700410Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'5', u'versionId': u'', u'etag': u'8f2cfdea', u'sequencer': u'1E087F5E4BD8CD10', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.156 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383454.386706.0080e9b3', u'eventTime': u'2020-03-28 08:17:32.292542Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'2', u'versionId': u'', u'etag': u'0080e9b3', u'sequencer': u'1E087F5E5EAA0C17', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.156 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383454.498880.e2b225b0', u'eventTime': u'2020-03-28 08:17:33.298689Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'8', u'versionId': u'', u'etag': u'e2b225b0', u'sequencer': u'1E087F5ED04EBC1D', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.156 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383454.867064.52045da5', u'eventTime': u'2020-03-28 08:17:33.052502Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'7', u'versionId': u'', u'etag': u'52045da5', u'sequencer': u'1E087F5EB65AAE33', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.156 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383454.868468.39f3e527', u'eventTime': u'2020-03-28 08:17:32.161612Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'1', u'versionId': u'', u'etag': u'39f3e527', u'sequencer': u'1E087F5E8BC6C333', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.156 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383456.504153.e2b225b0', u'eventTime': u'2020-03-28 08:17:33.298689Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'8', u'versionId': u'', u'etag': u'e2b225b0', u'sequencer': u'20087F5E28C50C1E', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.157 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383456.579287.52045da5', u'eventTime': u'2020-03-28 08:17:33.052502Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'7', u'versionId': u'', u'etag': u'52045da5', u'sequencer': u'20087F5E913A8722', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.157 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383456.781265.e2b225b0', u'eventTime': u'2020-03-28 08:17:33.298689Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'8', u'versionId': u'', u'etag': u'e2b225b0', u'sequencer': u'20087F5EDC2B912E', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.157 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383457.305138.e2b225b0', u'eventTime': u'2020-03-28 08:17:33.298689Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'8', u'versionId': u'', u'etag': u'e2b225b0', u'sequencer': u'21087F5E880C3012', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.157 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383457.350975.2f8bffda', u'eventTime': u'2020-03-28 08:17:33.619334Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'9', u'versionId': u'', u'etag': u'2f8bffda', u'sequencer': u'21087F5EE777EB14', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.157 DEBUG:tasks.rgw_multi.tests_ps:{u'eventVersion': u'2.2', u'eventId': u'1585383458.167601.2f8bffda', u'eventTime': u'2020-03-28 08:17:33.619334Z', u'requestParameters': {u'sourceIPAddress': u''}, u's3': {u'configurationId': u'axxsjh-71_notif', u'object': {u'key': u'9', u'versionId': u'', u'etag': u'2f8bffda', u'sequencer': u'22087F5E3C65FD09', u'size': 0, u'metadata': []}, u'bucket': {u'arn': u'arn:aws:s3:::axxsjh-71', u'name': u'axxsjh-71', u'ownerIdentity': {u'principalId': u'rgw-multisite-test-user'}, u'id': u'5fb75e0a-cb77-479b-b462-3a1befb8717c.5445.22'}, u's3SchemaVersion': u'1.0'}, u'responseElements': {u'x-amz-id-2': u'', u'x-amz-request-id': u''}, u'awsRegion': u'', u'eventName': u's3:ObjectCreated:*', u'userIdentity': {u'principalId': u''}, u'eventSource': u'ceph:s3'} 2020-03-28T08:17:40.260 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2020-03-28T08:17:43.929 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2020-03-28T08:17:43.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:43.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:43.930 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:45.472 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:45.791 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383463.765745_225.1","last_update":"2020-03-28 08:17:43.765745Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:45.791 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:17:45.792 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:45.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:45.792 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:47.129 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383463.765745_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:43.765745Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:47.130 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:17:47.130 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:17:47.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:47.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:47.131 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:48.637 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383463.765745_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:43.765745Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:48.638 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:17:48.638 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:48.638 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:48.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:48.639 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:51.341 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383463.765745_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:43.765745Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:51.342 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:51.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:51.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:51.343 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:52.603 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:52.892 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383463.765745_225.1","last_update":"2020-03-28 08:17:43.765745Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:52.893 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:52.893 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:52.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:52.894 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:55.184 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383463.765745_225.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:43.765745Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:55.184 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:17:55.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:55.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:55.716 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:56.910 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:17:57.196 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383475.618524_229.1","last_update":"2020-03-28 08:17:55.618524Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:17:57.196 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:17:57.196 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:57.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:17:57.197 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:17:59.666 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383475.618524_229.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:55.618524Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:17:59.667 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:00.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:00.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:00.369 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:01.565 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:01.923 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383475.618524_229.1","last_update":"2020-03-28 08:17:55.618524Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:01.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:01.923 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:01.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:01.924 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:04.516 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383475.618524_229.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:17:55.618524Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:04.516 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:05.751 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2020-03-28T08:18:05.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:05.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:05.752 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:06.799 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:07.086 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383485.682131_231.1","last_update":"2020-03-28 08:18:05.682131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:07.086 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:18:07.087 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:07.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:07.087 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:08.601 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:08.601 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:18:08.601 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:18:08.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:08.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:08.602 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:08.883 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:00:40.854 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:18:08.894 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:00:40.865 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:18:09.104 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:00:41.075 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:18:11.658 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:11.658 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:18:11.658 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:11.658 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:11.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:11.659 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:14.224 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:14.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:14.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:14.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:14.225 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:15.318 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:15.661 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383485.682131_231.1","last_update":"2020-03-28 08:18:05.682131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:15.661 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:15.661 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:15.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:15.662 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:18.445 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:18.445 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:20.624 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2020-03-28T08:18:20.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:20.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:20.625 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:21.660 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:22.046 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383485.682131_231.1","last_update":"2020-03-28 08:18:05.682131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:22.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:18:22.046 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:22.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:22.047 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:23.501 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:23.501 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:18:23.502 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:18:23.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:23.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:23.502 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:24.969 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:24.970 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:18:24.970 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:24.970 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:24.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:24.971 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:27.192 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:27.193 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:27.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:27.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:27.193 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:28.366 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:28.644 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383485.682131_231.1","last_update":"2020-03-28 08:18:05.682131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:28.645 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:28.645 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:28.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:28.646 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:31.034 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:31.035 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:31.035 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2020-03-28T08:18:31.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:31.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:31.036 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:32.153 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:32.479 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383485.682131_231.1","last_update":"2020-03-28 08:18:05.682131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:32.480 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:18:32.480 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:32.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:32.481 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:34.156 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:34.157 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:18:34.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:18:34.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:34.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:34.158 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:35.630 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:35.631 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:18:35.631 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:35.631 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:35.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:35.632 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:38.029 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:38.029 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:38.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:38.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:38.030 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:39.122 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:39.481 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383485.682131_231.1","last_update":"2020-03-28 08:18:05.682131Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:39.482 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:39.484 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:39.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:39.484 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:41.989 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383485.682131_231.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:05.682131Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:41.990 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:45.259 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-03-28T08:18:45.260 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2020-03-28T08:18:45.260 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2020-03-28T08:18:45.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:45.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:45.261 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:46.313 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:46.584 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383525.188634_237.1","last_update":"2020-03-28 08:18:45.188634Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:46.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:18:46.585 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:46.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:46.585 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:48.051 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383525.188634_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:45.188634Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:48.052 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:18:48.052 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:18:48.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:48.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:48.053 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:49.426 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383525.188634_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:45.188634Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:49.426 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:18:49.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:49.426 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:49.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:49.427 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:51.880 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383525.188634_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:45.188634Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:51.881 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:51.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:51.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:51.881 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:53.002 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:53.331 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383525.188634_237.1","last_update":"2020-03-28 08:18:45.188634Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:53.331 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:18:53.331 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:53.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:53.332 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:55.822 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383525.188634_237.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:45.188634Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:18:55.823 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:18:58.361 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-03-28T08:18:58.361 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2020-03-28T08:18:58.362 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2020-03-28T08:18:58.362 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-03-28T08:18:58.362 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-03-28T08:18:58.362 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2020-03-28T08:18:58.362 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2020-03-28T08:18:58.363 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2020-03-28T08:18:58.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:58.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:58.363 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:18:58.931 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:00:24.991 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:18:58.937 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:00:24.997 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:18:59.147 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:00:25.207 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:18:59.543 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:18:59.854 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383538.264957_243.1","last_update":"2020-03-28 08:18:58.264957Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:18:59.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:18:59.855 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:59.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:18:59.856 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:01.450 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:01.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:19:01.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:19:01.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:01.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:01.452 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:03.070 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:03.070 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:19:03.070 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:03.070 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:03.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:03.071 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:05.315 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:05.316 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:05.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:05.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:05.317 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:06.373 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:06.680 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383538.264957_243.1","last_update":"2020-03-28 08:18:58.264957Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:06.680 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:06.681 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:06.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:06.681 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:08.972 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:08.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:09.572 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2020-03-28T08:19:09.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:09.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:09.573 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:10.690 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:11.003 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383538.264957_243.1","last_update":"2020-03-28 08:18:58.264957Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:11.003 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:19:11.003 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:11.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:11.004 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:12.369 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:12.370 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:19:12.370 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:19:12.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:12.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:12.371 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:13.835 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:13.835 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:19:13.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:13.836 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:13.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:13.836 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:16.343 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:16.344 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:16.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:16.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:16.344 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:17.403 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:17.726 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383538.264957_243.1","last_update":"2020-03-28 08:18:58.264957Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:17.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:17.727 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:17.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:17.727 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:20.381 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:20.382 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:20.945 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2020-03-28T08:19:20.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:20.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:20.946 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:22.346 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:22.634 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383538.264957_243.1","last_update":"2020-03-28 08:18:58.264957Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:22.634 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:19:22.634 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:22.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:22.635 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:24.090 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:24.090 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:19:24.091 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:19:24.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:24.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:24.091 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:25.545 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:25.545 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:19:25.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:25.545 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:25.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:25.546 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:28.016 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:28.017 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:28.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:28.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:28.018 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:29.161 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:29.593 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383538.264957_243.1","last_update":"2020-03-28 08:18:58.264957Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:29.593 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:29.593 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:29.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:29.594 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:31.978 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383538.264957_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:18:58.264957Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:31.979 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:32.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:32.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:32.758 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:33.915 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:34.213 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383572.621926_247.1","last_update":"2020-03-28 08:19:32.621926Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:34.214 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:34.214 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:34.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:34.215 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:36.656 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383572.621926_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:32.621926Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:36.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:37.902 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2020-03-28T08:19:37.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:37.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:37.904 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:39.054 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:39.337 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383577.815355_249.1","last_update":"2020-03-28 08:19:37.815355Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:39.337 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:19:39.337 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:39.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:39.338 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:40.715 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383577.815355_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:37.815355Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:40.715 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:19:40.715 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:19:40.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:40.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:40.716 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:42.114 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383577.815355_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:37.815355Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:42.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:19:42.114 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:42.115 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:42.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:42.115 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:44.524 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383577.815355_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:37.815355Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:44.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:44.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:44.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:44.528 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:45.628 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:45.929 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383577.815355_249.1","last_update":"2020-03-28 08:19:37.815355Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:45.929 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:45.929 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:45.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:45.930 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:48.288 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383577.815355_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:37.815355Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:48.288 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:49.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:49.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:49.061 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:50.117 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:50.548 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383588.989773_253.1","last_update":"2020-03-28 08:19:48.989773Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:50.548 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:50.549 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:50.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:50.549 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:53.027 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383588.989773_253.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:48.989773Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:53.028 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:19:54.105 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2020-03-28T08:19:54.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:54.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:54.106 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:55.264 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:19:55.543 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383594.004733_255.1","last_update":"2020-03-28 08:19:54.004733Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:19:55.543 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:19:55.543 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:55.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:55.544 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:57.079 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383594.004733_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:54.004733Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:57.080 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:19:57.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:19:57.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:57.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:57.081 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:19:58.519 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383594.004733_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:54.004733Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:19:58.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:19:58.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:19:58.520 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:58.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:19:58.520 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:01.095 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383594.004733_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:54.004733Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:01.096 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:20:01.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:01.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:01.097 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:02.296 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:20:02.589 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383594.004733_255.1","last_update":"2020-03-28 08:19:54.004733Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:20:02.589 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:20:02.590 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:02.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:02.590 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:05.169 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383594.004733_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:19:54.004733Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:05.170 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:20:06.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:06.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:06.097 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:07.305 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:20:07.630 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383605.993502_259.1","last_update":"2020-03-28 08:20:05.993502Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:20:07.630 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:20:07.630 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:07.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:07.631 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:10.097 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383605.993502_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:20:05.993502Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:10.097 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:20:13.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-80', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:13.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-80', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:13.795 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-80 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:17.408 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000022.30.3"}]} 2020-03-28T08:20:17.409 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-80 2020-03-28T08:20:17.409 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-80', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:17.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-80', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:17.409 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-80 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:19.861 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"9","instance":"","ns":""},"count":10},"inc_marker":{"position":"00000000022.30.3"}}}] 2020-03-28T08:20:19.862 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-80 2020-03-28T08:20:20.488 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2020-03-28T08:20:20.488 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2020-03-28T08:20:20.488 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:20:20.489 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:20:20.490 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:20:20.490 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:20:20.490 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:20:20.609 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:20:22.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:22.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:22.544 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:24.974 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:20:25.361 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383605.993502_259.1","last_update":"2020-03-28 08:20:05.993502Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:20:25.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:20:25.361 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:25.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:25.362 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:27.845 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383605.993502_259.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:20:05.993502Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:27.845 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:20:27.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-80', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:27.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-80', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:27.846 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-80 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:29.312 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000042.50.5"}]} 2020-03-28T08:20:29.312 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-80 2020-03-28T08:20:29.312 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-80', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:29.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-80', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:29.313 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-80 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:31.597 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"9","instance":"","ns":""},"count":10},"inc_marker":{"position":"00000000042.50.5"}}}] 2020-03-28T08:20:31.598 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-80 2020-03-28T08:20:32.398 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2020-03-28T08:20:32.398 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2020-03-28T08:20:32.399 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:20:32.400 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:20:32.400 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:20:32.400 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:20:32.400 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:20:33.407 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2020-03-28T08:20:33.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:33.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:33.408 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:34.684 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:20:35.014 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383633.322586_261.1","last_update":"2020-03-28 08:20:33.322586Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:20:35.140 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:20:35.141 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:35.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:35.141 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:36.603 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383633.322586_261.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:20:33.322586Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:36.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:20:36.604 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:20:36.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:36.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:36.604 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:38.041 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383633.322586_261.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:20:33.322586Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:38.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:20:38.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:20:38.042 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:38.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:38.042 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:40.452 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383633.322586_261.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:20:33.322586Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:40.453 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:20:40.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:40.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:40.454 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:41.520 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:20:41.831 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383633.322586_261.1","last_update":"2020-03-28 08:20:33.322586Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:20:41.831 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:20:41.831 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:41.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:41.832 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:44.342 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383633.322586_261.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:20:33.322586Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:44.342 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:20:45.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:45.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:45.408 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:46.440 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:20:46.728 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383645.315250_265.1","last_update":"2020-03-28 08:20:45.315250Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:20:46.728 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:20:46.728 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:46.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:46.729 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:49.353 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383645.315250_265.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:20:45.315250Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:20:49.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:20:49.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:49.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:49.354 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-81 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:50.914 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":""}]} 2020-03-28T08:20:50.915 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-81 2020-03-28T08:20:50.915 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:50.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:50.915 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-81 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:20:53.224 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:20:53.224 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-81 2020-03-28T08:20:59.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:59.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:20:59.101 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-81 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:01.875 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000022.28.5"}]} 2020-03-28T08:21:01.876 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-81 2020-03-28T08:21:01.876 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:01.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:01.877 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-81 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:05.067 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"6","instance":"","ns":""},"count":7},"inc_marker":{"position":"00000000022.28.5"}}}] 2020-03-28T08:21:05.068 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-81 2020-03-28T08:21:05.562 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2020-03-28T08:21:05.562 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:05.562 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:21:05.563 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:21:05.808 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:21:06.421 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2020-03-28T08:21:06.421 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:06.421 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2020-03-28T08:21:06.421 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2020-03-28T08:21:06.421 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2020-03-28T08:21:06.421 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2020-03-28T08:21:06.422 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2020-03-28T08:21:06.422 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:06.422 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2020-03-28T08:21:06.422 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:06.422 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:21:06.422 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:21:06.422 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:21:09.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:09.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-81', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:09.088 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-81 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:10.828 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000042.48.5"}]} 2020-03-28T08:21:10.828 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-81 2020-03-28T08:21:10.829 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:10.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:10.829 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-81 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:13.288 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"6","instance":"","ns":""},"count":7},"inc_marker":{"position":"00000000039.45.4"}}}] 2020-03-28T08:21:13.288 WARNING:rgw_multi.tests:bucket axxsjh-81 zone test-zone4 behind zone test-zone1: shard=0 master=00000000042.48.5 target=00000000039.45.4 2020-03-28T08:21:18.289 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:18.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:18.289 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-81 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:20.816 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"6","instance":"","ns":""},"count":7},"inc_marker":{"position":"00000000039.45.4"}}}] 2020-03-28T08:21:20.817 WARNING:rgw_multi.tests:bucket axxsjh-81 zone test-zone4 behind zone test-zone1: shard=0 master=00000000042.48.5 target=00000000039.45.4 2020-03-28T08:21:25.817 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:25.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:25.818 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-81 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:28.459 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"6","instance":"","ns":""},"count":7},"inc_marker":{"position":"00000000039.45.4"}}}] 2020-03-28T08:21:28.460 WARNING:rgw_multi.tests:bucket axxsjh-81 zone test-zone4 behind zone test-zone1: shard=0 master=00000000042.48.5 target=00000000039.45.4 2020-03-28T08:21:33.461 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:33.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-81', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:33.462 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-81 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:36.095 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"6","instance":"","ns":""},"count":7},"inc_marker":{"position":"00000000042.48.5"}}}] 2020-03-28T08:21:36.095 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-81 2020-03-28T08:21:36.096 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE) synced 2020-03-28T08:21:36.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "0" type: "OBJECT_CREATE" 2020-03-28T08:21:36.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:36.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "1" type: "OBJECT_CREATE" 2020-03-28T08:21:36.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "3" type: "OBJECT_CREATE" 2020-03-28T08:21:36.544 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "5" type: "OBJECT_CREATE" 2020-03-28T08:21:36.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "2" type: "OBJECT_CREATE" 2020-03-28T08:21:36.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "4" type: "OBJECT_CREATE" 2020-03-28T08:21:36.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:36.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "7" type: "OBJECT_CREATE" 2020-03-28T08:21:36.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:36.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:21:36.545 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE): objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:21:36.546 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:21:36.965 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "0" type: "OBJECT_DELETE" 2020-03-28T08:21:36.965 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "2" type: "OBJECT_DELETE" 2020-03-28T08:21:36.965 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "4" type: "OBJECT_DELETE" 2020-03-28T08:21:36.965 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "1" type: "OBJECT_DELETE" 2020-03-28T08:21:36.965 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "3" type: "OBJECT_DELETE" 2020-03-28T08:21:36.965 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "5" type: "OBJECT_DELETE" 2020-03-28T08:21:36.965 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "6" type: "OBJECT_DELETE" 2020-03-28T08:21:36.966 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "7" type: "OBJECT_DELETE" 2020-03-28T08:21:36.966 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "8" type: "OBJECT_DELETE" 2020-03-28T08:21:36.966 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_DELETE): objname: "9" type: "OBJECT_DELETE" 2020-03-28T08:21:36.966 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:21:37.432 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "0" type: "OBJECT_CREATE" 2020-03-28T08:21:37.432 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:37.432 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "1" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "3" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "5" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "2" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "4" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "7" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:21:37.433 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:21:37.434 DEBUG:tasks.rgw_multi.tests_ps:Event (OBJECT_CREATE,OBJECT_DELETE): objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:21:37.434 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:21:39.725 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... ok 2020-03-28T08:21:39.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:39.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:39.726 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:40.733 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:21:41.016 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383699.612522_290.1","last_update":"2020-03-28 08:21:39.612522Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:21:41.016 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:21:41.016 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:41.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:41.017 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:42.439 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383699.612522_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:21:39.612522Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:21:42.439 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:21:42.440 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:21:42.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:42.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:42.440 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:44.010 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383699.612522_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:21:39.612522Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:21:44.010 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:21:44.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:21:44.011 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:44.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:44.011 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:46.495 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383699.612522_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:21:39.612522Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:21:46.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:21:46.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:46.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:46.497 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:47.637 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:21:47.943 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383699.612522_290.1","last_update":"2020-03-28 08:21:39.612522Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:21:47.943 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:21:47.944 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:47.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:47.944 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:50.370 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383699.612522_290.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:21:39.612522Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:21:50.370 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:21:51.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:51.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:51.212 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:52.364 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:21:52.752 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383711.127491_294.1","last_update":"2020-03-28 08:21:51.127491Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:21:52.752 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:21:52.753 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:52.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:21:52.753 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:21:55.334 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383711.127491_294.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:21:51.127491Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:21:55.335 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:22:53.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:22:53.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:22:53.279 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-82 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:22:55.752 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000200.217.5"}]} 2020-03-28T08:22:55.752 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-82 2020-03-28T08:22:55.752 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:22:55.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:22:55.753 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-82 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:22:58.455 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"8","instance":"","ns":""},"count":9},"inc_marker":{"position":"00000000200.217.5"}}}] 2020-03-28T08:22:58.455 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-82 2020-03-28T08:22:59.103 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2020-03-28T08:22:59.104 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:22:59.104 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2020-03-28T08:22:59.104 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2020-03-28T08:22:59.104 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2020-03-28T08:22:59.104 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2020-03-28T08:22:59.104 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2020-03-28T08:22:59.104 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2020-03-28T08:22:59.105 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2020-03-28T08:22:59.105 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-03-28T08:22:59.105 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2020-03-28T08:22:59.105 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2020-03-28T08:22:59.105 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2020-03-28T08:22:59.105 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-03-28T08:22:59.105 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2020-03-28T08:22:59.860 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2020-03-28T08:22:59.860 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2020-03-28T08:22:59.860 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "20" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "21" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "22" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "23" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "24" type: "OBJECT_CREATE" 2020-03-28T08:22:59.861 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "29" type: "OBJECT_CREATE" 2020-03-28T08:22:59.862 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "27" type: "OBJECT_CREATE" 2020-03-28T08:22:59.862 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "25" type: "OBJECT_CREATE" 2020-03-28T08:22:59.862 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "26" type: "OBJECT_CREATE" 2020-03-28T08:23:00.454 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "28" type: "OBJECT_CREATE" 2020-03-28T08:23:00.455 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "30" type: "OBJECT_CREATE" 2020-03-28T08:23:00.455 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "31" type: "OBJECT_CREATE" 2020-03-28T08:23:00.455 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "32" type: "OBJECT_CREATE" 2020-03-28T08:23:00.455 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "33" type: "OBJECT_CREATE" 2020-03-28T08:23:00.455 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "34" type: "OBJECT_CREATE" 2020-03-28T08:23:00.455 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "35" type: "OBJECT_CREATE" 2020-03-28T08:23:00.455 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "36" type: "OBJECT_CREATE" 2020-03-28T08:23:00.456 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "37" type: "OBJECT_CREATE" 2020-03-28T08:23:00.456 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "41" type: "OBJECT_CREATE" 2020-03-28T08:23:00.456 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "42" type: "OBJECT_CREATE" 2020-03-28T08:23:00.456 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "39" type: "OBJECT_CREATE" 2020-03-28T08:23:00.456 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "40" type: "OBJECT_CREATE" 2020-03-28T08:23:00.456 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "38" type: "OBJECT_CREATE" 2020-03-28T08:23:00.456 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "43" type: "OBJECT_CREATE" 2020-03-28T08:23:00.967 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "44" type: "OBJECT_CREATE" 2020-03-28T08:23:00.967 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "45" type: "OBJECT_CREATE" 2020-03-28T08:23:00.967 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "46" type: "OBJECT_CREATE" 2020-03-28T08:23:00.967 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "47" type: "OBJECT_CREATE" 2020-03-28T08:23:00.967 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "48" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "49" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "50" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "51" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "52" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "53" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "54" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "55" type: "OBJECT_CREATE" 2020-03-28T08:23:00.968 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "56" type: "OBJECT_CREATE" 2020-03-28T08:23:00.969 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "57" type: "OBJECT_CREATE" 2020-03-28T08:23:00.969 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "58" type: "OBJECT_CREATE" 2020-03-28T08:23:01.583 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "59" type: "OBJECT_CREATE" 2020-03-28T08:23:01.584 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "60" type: "OBJECT_CREATE" 2020-03-28T08:23:01.584 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "61" type: "OBJECT_CREATE" 2020-03-28T08:23:01.584 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "62" type: "OBJECT_CREATE" 2020-03-28T08:23:01.584 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "63" type: "OBJECT_CREATE" 2020-03-28T08:23:01.584 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "65" type: "OBJECT_CREATE" 2020-03-28T08:23:01.584 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "64" type: "OBJECT_CREATE" 2020-03-28T08:23:01.584 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "66" type: "OBJECT_CREATE" 2020-03-28T08:23:01.585 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "67" type: "OBJECT_CREATE" 2020-03-28T08:23:01.585 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "68" type: "OBJECT_CREATE" 2020-03-28T08:23:01.585 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "69" type: "OBJECT_CREATE" 2020-03-28T08:23:01.585 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "70" type: "OBJECT_CREATE" 2020-03-28T08:23:01.585 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "71" type: "OBJECT_CREATE" 2020-03-28T08:23:01.585 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "72" type: "OBJECT_CREATE" 2020-03-28T08:23:01.585 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "76" type: "OBJECT_CREATE" 2020-03-28T08:23:02.112 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "77" type: "OBJECT_CREATE" 2020-03-28T08:23:02.112 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "75" type: "OBJECT_CREATE" 2020-03-28T08:23:02.112 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "73" type: "OBJECT_CREATE" 2020-03-28T08:23:02.112 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "74" type: "OBJECT_CREATE" 2020-03-28T08:23:02.112 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "78" type: "OBJECT_CREATE" 2020-03-28T08:23:02.112 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "79" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "80" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "82" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "81" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "84" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "83" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "85" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "86" type: "OBJECT_CREATE" 2020-03-28T08:23:02.113 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "87" type: "OBJECT_CREATE" 2020-03-28T08:23:02.114 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "88" type: "OBJECT_CREATE" 2020-03-28T08:23:02.529 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "89" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "90" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "91" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "92" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "93" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "94" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "95" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "96" type: "OBJECT_CREATE" 2020-03-28T08:23:02.530 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "99" type: "OBJECT_CREATE" 2020-03-28T08:23:02.531 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "97" type: "OBJECT_CREATE" 2020-03-28T08:23:02.531 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "98" type: "OBJECT_CREATE" 2020-03-28T08:23:03.120 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:23:18.648 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:05:50.619 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:23:18.668 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:05:50.639 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:23:18.831 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:05:50.802 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:23:44.591 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... ok 2020-03-28T08:23:44.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:44.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:44.592 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:45.922 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:23:46.310 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383824.483516_296.1","last_update":"2020-03-28 08:23:44.483516Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:23:46.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:23:46.311 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:46.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:46.312 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:47.592 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383824.483516_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:23:44.483516Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:23:47.592 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:23:47.592 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:23:47.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:47.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:47.593 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:48.950 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383824.483516_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:23:44.483516Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:23:48.951 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:23:48.951 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:23:48.951 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:48.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:48.952 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:51.223 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383824.483516_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:23:44.483516Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:23:51.224 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:23:51.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:51.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:51.225 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:52.485 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:23:52.823 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383824.483516_296.1","last_update":"2020-03-28 08:23:44.483516Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:23:52.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:23:52.824 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:52.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:52.825 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:55.132 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383824.483516_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:23:44.483516Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:23:55.132 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:23:56.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:56.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:56.038 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:57.194 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:23:57.590 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383835.743614_300.1","last_update":"2020-03-28 08:23:55.743614Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:23:57.591 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:23:57.591 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:57.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:23:57.591 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:23:59.965 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383835.743614_300.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:23:55.743614Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:23:59.965 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:24:05.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:05.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-83', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:05.718 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-83 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:07.755 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000022.24.5"}]} 2020-03-28T08:24:07.755 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-83 2020-03-28T08:24:07.756 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-83', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:07.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-83', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:07.756 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-83 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:10.157 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"3","instance":"","ns":""},"count":4},"inc_marker":{"position":"00000000022.24.5"}}}] 2020-03-28T08:24:10.158 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-83 2020-03-28T08:24:10.769 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383842.013410.e6328b84" 2020-03-28T08:24:10.769 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383842.016693.60548721" 2020-03-28T08:24:10.769 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383842.037275.a9a6c64f" 2020-03-28T08:24:10.769 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383842.169625.b2f111d9" 2020-03-28T08:24:10.769 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383844.281155.a9a6c64f" 2020-03-28T08:24:10.769 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383844.438951.b2f111d9" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383844.840237.b2f111d9" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383844.939333.8f1cb5b9" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383845.062929.52f6d652" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383845.493311.52f6d652" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383845.625301.5f71f78f" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383845.891499.1ca64233" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383845.946590.fd566e36" 2020-03-28T08:24:10.770 DEBUG:tasks.rgw_multi.tests_ps:Event (before ack) id: "1585383846.698016.07eeda9a" 2020-03-28T08:24:10.973 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-03-28T08:24:12.266 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383844.438951.b2f111d9" 2020-03-28T08:24:12.266 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383844.840237.b2f111d9" 2020-03-28T08:24:12.266 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383844.939333.8f1cb5b9" 2020-03-28T08:24:12.266 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383845.062929.52f6d652" 2020-03-28T08:24:12.266 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383845.493311.52f6d652" 2020-03-28T08:24:12.267 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383845.625301.5f71f78f" 2020-03-28T08:24:12.267 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383845.891499.1ca64233" 2020-03-28T08:24:12.267 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383845.946590.fd566e36" 2020-03-28T08:24:12.267 DEBUG:tasks.rgw_multi.tests_ps:Event (after ack) id: "1585383846.698016.07eeda9a" 2020-03-28T08:24:16.553 INFO:tasks.rgw_multisite_tests:test acking of some events in a subscription ... ok 2020-03-28T08:24:16.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:16.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:16.554 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:18.115 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:24:18.475 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383856.450238_302.1","last_update":"2020-03-28 08:24:16.450238Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:24:18.476 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:24:18.476 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:18.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:18.477 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:19.800 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383856.450238_302.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:16.450238Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:19.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:24:19.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:24:19.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:19.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:19.802 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:21.169 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383856.450238_302.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:16.450238Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:21.170 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:24:21.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:24:21.170 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:21.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:21.171 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:23.801 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383856.450238_302.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:16.450238Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:23.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:24:23.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:23.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:23.802 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:24.945 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:24:25.279 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383856.450238_302.1","last_update":"2020-03-28 08:24:16.450238Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:24:25.279 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:24:25.280 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:25.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:25.280 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:25.614 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:05:51.674 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:24:25.626 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:05:51.686 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:24:25.866 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:05:51.926 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:24:28.059 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383856.450238_302.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:16.450238Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:28.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:24:28.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:28.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:28.854 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:29.952 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:24:30.255 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383868.755440_306.1","last_update":"2020-03-28 08:24:28.755440Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:24:30.256 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:24:30.256 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:30.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:30.256 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:32.543 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383868.755440_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:28.755440Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:32.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:24:35.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:35.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-84', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:35.754 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-84 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:37.961 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000013.56.5"}]} 2020-03-28T08:24:37.961 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-84 2020-03-28T08:24:37.962 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-84', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:37.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-84', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:37.962 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-84 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:40.702 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"put","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000013.56.5"}}}] 2020-03-28T08:24:40.702 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-84 2020-03-28T08:24:40.906 DEBUG:tasks.rgw_multi.tests_ps:Event key: "copy" type: "OBJECT_CREATE" 2020-03-28T08:24:40.906 DEBUG:tasks.rgw_multi.tests_ps:Event key: "put" type: "OBJECT_CREATE" 2020-03-28T08:24:40.907 DEBUG:tasks.rgw_multi.tests_ps:Event key: "multipart" type: "OBJECT_CREATE" 2020-03-28T08:24:42.325 INFO:tasks.rgw_multisite_tests:test object creation notifications in using put/copy/post ... ok 2020-03-28T08:24:42.325 INFO:tasks.rgw_multisite_tests:test object creation s3 notifications in using put/copy/post on master ... ok 2020-03-28T08:24:42.325 INFO:tasks.rgw_multisite_tests:test multipart object upload on master ... ok 2020-03-28T08:24:42.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:42.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:42.326 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:43.669 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:24:44.049 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383882.254498_308.1","last_update":"2020-03-28 08:24:42.254498Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:24:44.050 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:24:44.050 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:44.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:44.050 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:45.508 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383882.254498_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:42.254498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:45.509 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:24:45.509 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:24:45.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:45.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:45.509 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:46.815 INFO:teuthology.orchestra.run.smithi156.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383882.254498_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:42.254498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:46.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-03-28T08:24:46.816 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:24:46.816 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:46.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:46.816 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:49.331 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383882.254498_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:42.254498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:49.331 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:24:49.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:49.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:49.332 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:50.554 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:24:50.855 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383882.254498_308.1","last_update":"2020-03-28 08:24:42.254498Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:24:50.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:24:50.855 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:50.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:50.856 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:53.482 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383882.254498_308.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:42.254498Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:53.482 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:24:54.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:54.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:54.798 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:55.992 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:24:56.286 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383894.703881_314.1","last_update":"2020-03-28 08:24:54.703881Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:24:56.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-03-28T08:24:56.287 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:56.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:24:56.288 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:24:58.633 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383894.703881_314.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:54.703881Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:24:58.633 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-03-28T08:25:01.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:01.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:01.459 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:03.489 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:07:35.460 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:25:03.491 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:07:35.462 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:25:03.587 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:07:35.558 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:25:04.522 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000007.43.11"}]} 2020-03-28T08:25:04.523 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:04.523 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:04.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:04.523 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:06.937 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"foo","instance":"zF8juYho1hOgj1E1i1phZ19QpxsryWY","ns":""},"count":1},"inc_marker":{"position":"00000000007.43.11"}}}] 2020-03-28T08:25:06.937 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:08.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:08.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:08.463 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:10.535 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000014.59.5"}]} 2020-03-28T08:25:10.535 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:10.535 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:10.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:10.536 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:13.188 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"foo","instance":"zF8juYho1hOgj1E1i1phZ19QpxsryWY","ns":""},"count":1},"inc_marker":{"position":"00000000014.59.5"}}}] 2020-03-28T08:25:13.188 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:13.381 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2020-03-28T08:25:13.382 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2020-03-28T08:25:13.382 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "OBJECT_DELETE" 2020-03-28T08:25:13.520 DEBUG:tasks.rgw_multi.tests_ps:Event key: "foo" type: "DELETE_MARKER_CREATE" 2020-03-28T08:25:13.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:13.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:13.521 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:15.050 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000014.59.5"}]} 2020-03-28T08:25:15.050 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:15.051 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:15.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:15.051 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:16.653 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"foo","instance":"zF8juYho1hOgj1E1i1phZ19QpxsryWY","ns":""},"count":1},"inc_marker":{"position":"00000000014.59.5"}}}] 2020-03-28T08:25:16.654 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:16.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:16.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:16.654 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:18.583 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000014.59.5"}]} 2020-03-28T08:25:18.584 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:18.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:18.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:18.584 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:20.431 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"foo","instance":"zF8juYho1hOgj1E1i1phZ19QpxsryWY","ns":""},"count":3},"inc_marker":{"position":"00000000014.59.5"}}}] 2020-03-28T08:25:20.431 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:20.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:20.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:20.432 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:22.254 INFO:teuthology.orchestra.run.smithi156.stdout:{"markers":[{"key":0,"val":"00000000014.59.5"}]} 2020-03-28T08:25:22.254 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:22.254 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:22.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone1', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:22.255 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:24.967 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"foo","instance":"zF8juYho1hOgj1E1i1phZ19QpxsryWY","ns":""},"count":1},"inc_marker":{"position":"00000000014.59.5"}}}] 2020-03-28T08:25:24.967 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=axxsjh-85 2020-03-28T08:25:24.967 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:24.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:24.968 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:26.504 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000013.41.5"}]} 2020-03-28T08:25:26.505 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-85 2020-03-28T08:25:26.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:26.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:26.506 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:28.500 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"foo","instance":"zF8juYho1hOgj1E1i1phZ19QpxsryWY","ns":""},"count":3},"inc_marker":{"position":"00000000013.41.5"}}}] 2020-03-28T08:25:28.501 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=axxsjh-85 2020-03-28T08:25:28.501 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:28.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:28.501 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:29.963 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000013.41.5"}]} 2020-03-28T08:25:29.963 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-85 2020-03-28T08:25:29.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:29.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:29.964 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:31.850 INFO:teuthology.orchestra.run.smithi156.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"foo","instance":"zF8juYho1hOgj1E1i1phZ19QpxsryWY","ns":""},"count":3},"inc_marker":{"position":"00000000013.41.5"}}}] 2020-03-28T08:25:31.851 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=axxsjh-85 2020-03-28T08:25:31.851 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:31.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:31.852 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:33.535 INFO:teuthology.orchestra.run.smithi168.stdout:{"markers":[{"key":0,"val":"00000000013.41.5"}]} 2020-03-28T08:25:33.535 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone2 bucket=axxsjh-85 2020-03-28T08:25:33.535 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:33.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:33.536 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:36.059 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:25:36.060 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:25:41.061 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:41.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:41.061 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:43.403 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:25:43.404 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:25:48.404 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:48.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:48.405 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:50.961 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:25:50.962 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:25:53.430 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:07:19.490 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:25:53.454 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:07:19.514 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:25:53.618 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:07:19.678 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:25:55.963 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:55.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:25:55.964 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:25:58.496 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:25:58.496 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:03.497 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:03.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:03.498 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:06.103 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:06.103 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:11.104 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:11.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:11.105 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:13.691 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:13.692 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:18.692 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:18.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:18.693 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:21.210 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:21.210 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:26.211 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:26.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:26.211 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:28.780 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:28.781 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:33.781 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:33.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:33.782 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:36.366 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:36.367 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:41.368 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:41.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:41.369 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:43.838 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:43.838 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:48.839 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:48.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:48.840 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:51.504 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:51.504 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:26:56.504 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:56.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:26:56.505 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:26:58.996 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:26:58.996 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:03.997 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:03.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:03.998 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:27:06.679 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:27:06.679 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:11.681 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:11.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:11.681 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:27:14.503 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:27:14.503 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:19.504 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:19.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:19.505 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:27:22.045 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:27:22.045 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:27.046 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:27.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:27.047 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:27:29.711 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:27:29.711 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:34.712 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:34.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:34.714 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:27:37.339 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:27:37.339 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:42.340 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:42.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:42.341 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:27:45.230 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:27:45.231 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:50.232 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:50.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:50.233 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:27:52.828 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:27:52.829 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:27:57.830 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:57.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:27:57.830 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:00.419 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:00.419 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:05.420 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:05.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:05.421 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:08.180 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:08.181 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:13.182 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:13.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:13.183 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:16.064 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:16.065 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:21.066 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:21.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:21.067 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:23.914 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:23.915 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:28.916 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:28.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:28.916 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:31.501 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:31.501 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:36.503 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:36.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:36.503 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:38.765 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:38.766 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:43.767 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:43.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:43.767 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:46.159 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:46.159 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:51.160 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:51.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:51.161 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:28:53.793 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:28:53.793 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:28:58.794 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:58.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:28:58.795 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:01.509 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:01.509 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:29:06.510 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:06.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:06.511 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:09.051 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:09.051 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:29:14.053 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:14.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:14.053 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:17.093 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:17.093 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:29:22.094 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:22.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:22.094 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:24.828 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:24.829 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:29:29.830 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:29.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:29.830 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:32.716 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:32.717 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:29:37.717 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:37.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:37.718 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:40.389 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:40.390 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:29:45.391 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:45.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:45.392 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:48.076 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:48.076 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:29:53.077 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:53.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:29:53.078 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:29:55.790 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:29:55.791 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:00.792 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:00.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:00.793 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:03.298 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:03.299 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:08.300 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:08.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:08.301 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:10.967 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:10.968 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:15.968 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:15.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:15.969 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:18.602 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:18.603 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:23.605 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:23.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:23.605 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:26.428 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:26.429 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:31.430 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:31.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:31.431 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:34.008 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:34.008 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:39.010 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:39.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:39.011 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:41.579 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:41.580 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:46.580 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:46.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:46.581 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:49.413 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:49.414 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:30:54.415 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:54.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:30:54.415 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:30:57.200 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:30:57.201 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:02.202 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:02.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:02.202 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:04.696 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:04.696 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:09.698 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:09.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:09.698 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:12.483 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:12.484 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:17.485 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:17.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:17.486 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:20.122 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:20.123 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:25.124 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:25.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:25.125 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:28.269 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:28.269 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:33.270 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:33.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:33.271 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:35.981 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:35.982 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:39.599 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:14:11.569 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:31:39.605 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:14:11.576 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:31:39.849 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:14:11.820 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:31:40.983 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:40.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:40.983 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:43.912 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:43.912 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:48.914 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:48.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:48.914 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:51.586 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:51.587 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:56.588 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:56.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:31:56.588 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:31:59.017 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:14:30.988 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:31:59.032 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:14:31.003 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:31:59.051 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:31:59.051 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:31:59.108 INFO:tasks.ceph.c1.mon.a.smithi156.stderr:==00:02:14:31.078 8058== Warning: unimplemented fcntl command: 1036 2020-03-28T08:32:04.052 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:04.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:04.053 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:32:06.759 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:32:06.759 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:32:11.761 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:11.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:11.761 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:32:14.089 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:32:14.090 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:32:19.091 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:19.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:19.091 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:32:21.742 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:32:21.742 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:32:26.743 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:26.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:26.744 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:32:29.460 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:32:29.460 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:32:34.461 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:34.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:34.461 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:32:37.192 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:32:37.192 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:32:42.195 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:42.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:42.196 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:32:44.953 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:32:44.953 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:32:46.465 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:14:12.524 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:32:46.468 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:14:12.527 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:32:46.727 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:14:12.786 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:32:49.777 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:14:15.837 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:32:49.785 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:14:15.844 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:32:49.860 INFO:tasks.ceph.c2.mon.a.smithi168.stderr:==00:02:14:15.919 8269== Warning: unimplemented fcntl command: 1036 2020-03-28T08:32:49.954 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:49.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:49.954 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:32:52.856 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:32:52.856 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:32:57.858 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:57.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:32:57.858 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:33:00.810 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:33:00.810 WARNING:rgw_multi.tests:bucket axxsjh-85 zone test-zone4 behind zone test-zone2: shard=0 master=00000000013.41.5 target= 2020-03-28T08:33:05.812 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:05.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', u'test-zone4', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', u'test-zone2', '--bucket', 'axxsjh-85', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:05.812 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket axxsjh-85 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:33:07.567 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-28T08:33:07.567 DEBUG:teuthology.task.console_log:Killing console logger for smithi168 2020-03-28T08:33:07.568 DEBUG:tasks.rgw_multi.tests_ps:zonegroup_bucket_checkpoint failed, cannot delete bucket 2020-03-28T08:33:08.453 INFO:teuthology.orchestra.run.smithi168.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2020-03-28T08:33:08.792 INFO:tasks.rgw_multisite_tests:test notification of deletion markers ... ok 2020-03-28T08:33:08.792 INFO:tasks.rgw_multisite_tests:test s3 notification of metadata on master ... ok 2020-03-28T08:33:08.792 INFO:tasks.rgw_multisite_tests:test s3 notification of deletion markers on master ... ok 2020-03-28T08:33:08.793 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint ... ok 2020-03-28T08:33:08.793 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint s3 record format ... ok 2020-03-28T08:33:08.793 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint ... ok 2020-03-28T08:33:08.793 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint s3 record format ... ok 2020-03-28T08:33:08.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:08.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:08.794 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:33:10.049 INFO:teuthology.orchestra.run.smithi156.stderr:No --period given, using current period=0d07b3db-af62-4c83-b2b9-6aa7b82de92e 2020-03-28T08:33:10.414 INFO:teuthology.orchestra.run.smithi156.stdout:[{"marker":"1_1585383894.703881_314.1","last_update":"2020-03-28 08:24:54.703881Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-03-28T08:33:10.415 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-03-28T08:33:10.415 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:10.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone2', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:10.416 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:33:11.815 INFO:teuthology.orchestra.run.smithi168.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0d07b3db-af62-4c83-b2b9-6aa7b82de92e","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1585383894.703881_314.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-03-28 08:24:54.703881Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1585376798.437614_173.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-03-28 06:26:38.437614Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-03-28T08:33:11.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-03-28T08:33:11.816 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-03-28T08:33:11.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:11.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', u'test-zone3', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:11.817 INFO:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-03-28T08:33:12.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:33:12.826 INFO:tasks.rgw_multisite_tests:test notification status upon bucket deletion ... FAIL 2020-03-28T08:33:12.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:12.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', u'test-zone1', '--rgw-zonegroup', u'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-03-28T08:33:12.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:33:12.831 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:33:12.832 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:33:12.839 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... ERROR 2020-03-28T08:33:12.839 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok 2020-03-28T08:33:12.840 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok 2020-03-28T08:33:12.840 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok 2020-03-28T08:33:12.840 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:33:12.840 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:33:12.840 INFO:tasks.rgw_multisite_tests:ERROR: test creating a subscription when no topic info exists 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests_ps.py", line 3054, in test_ps_missing_topic 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests: zones, ps_zones = init_env() 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests_ps.py", line 559, in init_env 2020-03-28T08:33:12.841 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 206, in zonegroup_meta_checkpoint 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 152, in meta_master_log_status 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multisite.py", line 200, in admin 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/util/rgw.py", line 34, in rgwadmin 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: stdin=stdin, 2020-03-28T08:33:12.842 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests: raise ConnectionLostError(command=quote(args), node=name) 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests:ConnectionLostError: SSH connection to smithi156 was lost: u'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false' 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion 2020-03-28T08:33:12.843 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests_ps.py", line 2985, in test_ps_delete_bucket 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests: zones, ps_zones = init_env() 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests_ps.py", line 559, in init_env 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-03-28T08:33:12.844 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 211, in zonegroup_meta_checkpoint 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 188, in zone_meta_checkpoint 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests: period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multi/tests.py", line 145, in meta_sync_status 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests: assert(retcode == 2) # ENOENT 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests:AssertionError 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:33:12.845 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-03-28T08:33:12.846 INFO:tasks.rgw_multisite_tests:Ran 40 tests in 983.158s 2020-03-28T08:33:12.846 INFO:tasks.rgw_multisite_tests: 2020-03-28T08:33:12.846 INFO:tasks.rgw_multisite_tests:FAILED (errors=1, failures=1) 2020-03-28T08:33:12.846 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 125, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2020-03-28T08:33:12.875 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=2a5d0f36b58646a699bc35b5e68ea5eb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 125, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2020-03-28T08:33:12.875 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2020-03-28T08:33:12.883 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2020-03-28T08:33:12.899 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2020-03-28T08:33:12.928 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw.py", line 357, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 125, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2020-03-28T08:33:12.930 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-03-28T08:33:12.930 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:33:12.977 INFO:tasks.rgw.c2.client.1.smithi168.stdout:2020-03-28 08:33:12.934 1b85f700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 14853) UID: 0 2020-03-28T08:33:12.979 INFO:tasks.rgw.c2.client.1.smithi168.stdout:2020-03-28 08:33:12.943 4049040 -1 shutting down 2020-03-28T08:33:19.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:33:19.033 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw.py", line 357, in task yield File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw.py", line 156, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi168 was lost: u"cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.client.1.log --time-stamp=yes --vgdb=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1" 2020-03-28T08:33:19.037 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2020-03-28T08:33:19.048 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2020-03-28T08:33:19.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:33:19.048 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:19.050 DEBUG:teuthology.orchestra.remote:timed out 2020-03-28T08:34:19.050 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph.py", line 1897, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph_manager.py", line 1806, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph_manager.py", line 2018, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph_manager.py", line 1159, in raw_cluster_cmd stdout=BytesIO(), File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi168 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c2 osd dump --format=json' 2020-03-28T08:34:19.058 INFO:teuthology.misc:Shutting down mds daemons... 2020-03-28T08:34:19.059 INFO:teuthology.misc:Shutting down osd daemons... 2020-03-28T08:34:19.059 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2020-03-28T08:34:19.059 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:19.059 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:19.060 ERROR:teuthology.misc:Saw exception from c2.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi168 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.osd.1.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c2 -i 1" 2020-03-28T08:34:19.060 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2020-03-28T08:34:19.060 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:19.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:19.061 ERROR:teuthology.misc:Saw exception from c2.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi168 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.osd.0.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c2 -i 0" 2020-03-28T08:34:19.061 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2020-03-28T08:34:19.061 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:19.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:19.062 ERROR:teuthology.misc:Saw exception from c2.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi168 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.osd.2.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c2 -i 2" 2020-03-28T08:34:19.062 INFO:teuthology.misc:Shutting down mgr daemons... 2020-03-28T08:34:19.062 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2020-03-28T08:34:19.062 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:19.063 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:19.063 ERROR:teuthology.misc:Saw exception from c2.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi168 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster c2 -i x' 2020-03-28T08:34:19.063 INFO:teuthology.misc:Shutting down mon daemons... 2020-03-28T08:34:19.063 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2020-03-28T08:34:19.063 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:19.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:19.064 ERROR:teuthology.misc:Saw exception from c2.mon.a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi168 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c2.mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster c2 -i a" 2020-03-28T08:34:19.064 INFO:tasks.ceph:Checking cluster log for badness... 2020-03-28T08:34:19.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:19.065 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi168.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:22.133 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.168 2020-03-28T08:34:22.133 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph.py", line 1917, in task '--no-mon-health-to-clog', File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph.py", line 1128, in cluster config['log_whitelist']) is not None: File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph.py", line 1124, in first_in_ceph_log stdout = mon0_remote.sh(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 245, in sh proc=self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi168 was lost: "sudo egrep '\\[ERR\\]|\\[WRN\\]|\\[SEC\\]' /var/log/ceph/c2.log | egrep -v '\\(MDS_ALL_DOWN\\)' | egrep -v '\\(MDS_UP_LESS_THAN_MAX\\)' | head -n 1" 2020-03-28T08:34:22.134 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2020-03-28T08:34:22.145 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2020-03-28T08:34:22.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:22.145 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:40.469 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:34:40.470 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph.py", line 1897, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph_manager.py", line 1806, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph_manager.py", line 2018, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph_manager.py", line 1159, in raw_cluster_cmd stdout=BytesIO(), File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster c1 osd dump --format=json' 2020-03-28T08:34:40.471 INFO:teuthology.misc:Shutting down mds daemons... 2020-03-28T08:34:40.471 INFO:teuthology.misc:Shutting down osd daemons... 2020-03-28T08:34:40.471 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2020-03-28T08:34:40.471 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:40.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:40.472 ERROR:teuthology.misc:Saw exception from c1.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi156 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.osd.1.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c1 -i 1" 2020-03-28T08:34:40.472 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2020-03-28T08:34:40.472 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:40.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:40.473 ERROR:teuthology.misc:Saw exception from c1.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi156 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.osd.0.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c1 -i 0" 2020-03-28T08:34:40.473 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2020-03-28T08:34:40.473 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:40.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:40.474 ERROR:teuthology.misc:Saw exception from c1.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi156 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.osd.2.log --time-stamp=yes --vgdb=yes --tool=memcheck ceph-osd -f --cluster c1 -i 2" 2020-03-28T08:34:40.474 INFO:teuthology.misc:Shutting down mgr daemons... 2020-03-28T08:34:40.474 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2020-03-28T08:34:40.474 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:40.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:40.475 ERROR:teuthology.misc:Saw exception from c1.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi156 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster c1 -i x' 2020-03-28T08:34:40.475 INFO:teuthology.misc:Shutting down mon daemons... 2020-03-28T08:34:40.475 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2020-03-28T08:34:40.475 INFO:teuthology.orchestra.run:waiting for 300 2020-03-28T08:34:40.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-03-28T08:34:40.476 ERROR:teuthology.misc:Saw exception from c1.mon.a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1180, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status node=self.hostname) ConnectionLostError: SSH connection to smithi156 was lost: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/c1.mon.a.log --time-stamp=yes --vgdb=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster c1 -i a" 2020-03-28T08:34:40.476 INFO:tasks.ceph:Checking cluster log for badness... 2020-03-28T08:34:40.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:40.477 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:43.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:34:43.543 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2020-03-28T08:34:43.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:43.543 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:46.613 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:34:46.614 INFO:tasks.ceph:Archiving crash dumps... 2020-03-28T08:34:46.616 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252/remote/smithi156/crash 2020-03-28T08:34:46.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:46.617 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:49.685 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:34:49.685 INFO:tasks.ceph:Compressing logs... 2020-03-28T08:34:49.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:49.686 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:52.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:34:52.786 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph.py", line 1917, in task '--no-mon-health-to-clog', File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/ceph.py", line 267, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2020-03-28T08:34:52.787 DEBUG:teuthology.run_tasks:Unwinding manager install 2020-03-28T08:34:52.804 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 125, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-kefu2-testing-2020-03-27-2234/qa/tasks/rgw_multisite_tests.py", line 74, in begin raise RuntimeError(error_msg + 'test failures') RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2020-03-28T08:34:52.805 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-03-28T08:34:52.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:52.805 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:55.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:34:55.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:55.830 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:34:58.901 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:34:58.901 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task yield File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 218, in install remove_packages(ctx, config, package_list) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 104, in remove_packages system_type = teuthology.get_system_type(remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1198, in get_system_type system_value = remote.sh('sudo lsb_release -is').strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 245, in sh proc=self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'sudo lsb_release -is' 2020-03-28T08:34:58.902 DEBUG:teuthology.run_tasks:Unwinding manager clock 2020-03-28T08:34:58.913 INFO:teuthology.task.clock:Checking final clock skew... 2020-03-28T08:34:58.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:34:58.914 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:35:01.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:35:01.973 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/clock.py", line 65, in task 'true' File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true' 2020-03-28T08:35:01.976 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2020-03-28T08:35:01.988 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2020-03-28T08:35:01.988 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-03-28T08:35:02.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:35:02.002 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:35:05.045 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:35:05.045 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 138, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/selinux.py", line 157, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/selinux.py", line 188, in archive_log args=full_cmd.format(audit_archive=audit_archive) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' 2020-03-28T08:35:05.047 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2020-03-28T08:35:05.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2020-03-28T08:35:05.078 INFO:teuthology.task.internal:Duration was 9122.969878 seconds 2020-03-28T08:35:05.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2020-03-28T08:35:05.097 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-03-28T08:35:05.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:35:05.098 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:35:08.117 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:35:08.117 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/syslog.py", line 89, in syslog wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart' 2020-03-28T08:35:08.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2020-03-28T08:35:08.134 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2020-03-28T08:35:08.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:35:08.134 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:35:11.189 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:35:11.189 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 400, in sudo path=sudoers_file, ext=backup_ext File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers' 2020-03-28T08:35:11.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2020-03-28T08:35:11.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:35:11.204 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:35:14.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:35:14.261 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 443, in coredump wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump' 2020-03-28T08:35:14.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2020-03-28T08:35:14.275 INFO:teuthology.task.internal:Transferring archived files... 2020-03-28T08:35:14.275 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/kchai-2020-03-28_05:53:06-rgw-nautilus-kefu2-testing-2020-03-27-2234-distro-basic-smithi/4896252/remote/smithi156 2020-03-28T08:35:14.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:35:14.276 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:35:17.333 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:35:17.333 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 361, in archive misc.pull_directory(rem, archive_dir, path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 766, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 443, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .' 2020-03-28T08:35:17.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2020-03-28T08:35:17.348 INFO:teuthology.task.internal:Not uploading archives. 2020-03-28T08:35:17.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2020-03-28T08:35:17.363 INFO:teuthology.task.internal:Tidying up after the test... 2020-03-28T08:35:17.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-28T08:35:17.363 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi156.front.sepia.ceph.com', 'timeout': 60} 2020-03-28T08:35:20.405 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2020-03-28T08:35:20.406 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 49, in base wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 201, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 456, in run raise ConnectionLostError(command=quote(args), node=name) ConnectionLostError: SSH connection to smithi156 was lost: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2020-03-28T08:35:20.406 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2020-03-28T08:35:20.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines 2020-03-28T08:35:20.438 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: rgw multisite, rgw multisite pubsub, test failures 2020-03-28T08:35:20.438 INFO:teuthology.nuke:Checking targets against current locks 2020-03-28T08:35:20.545 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology", line 11, in load_entry_point('teuthology', 'console_scripts', 'teuthology')() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/scripts/run.py", line 34, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 392, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 253, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable